Jul 07 15:44:05.806161 ip-10-0-138-101 systemd[1]: Starting Kubernetes Kubelet...
Jul 07 15:44:06.228305 ip-10-0-138-101 kubenswrapper[2566]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 07 15:44:06.228305 ip-10-0-138-101 kubenswrapper[2566]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 07 15:44:06.228305 ip-10-0-138-101 kubenswrapper[2566]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 07 15:44:06.228305 ip-10-0-138-101 kubenswrapper[2566]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 07 15:44:06.228305 ip-10-0-138-101 kubenswrapper[2566]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 07 15:44:06.229466 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.229385    2566 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 07 15:44:06.236503 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.236475    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 15:44:06.236503 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.236498    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 15:44:06.236503 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.236502    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 15:44:06.236503 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.236505    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 15:44:06.236503 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.236509    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 15:44:06.236503 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.236512    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 15:44:06.236761 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.236515    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 15:44:06.236761 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.236518    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 15:44:06.236761 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.236521    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 15:44:06.236761 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.236523    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 15:44:06.236761 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.236526    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 15:44:06.236761 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.236529    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 15:44:06.236761 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.236532    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 15:44:06.236761 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.236535    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 15:44:06.236761 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.236537    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 15:44:06.236761 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.236540    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 15:44:06.236761 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.236543    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 15:44:06.236761 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.236546    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 15:44:06.236761 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.236548    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 15:44:06.236761 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.236551    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 15:44:06.236761 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.236553    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 15:44:06.236761 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.236556    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 15:44:06.236761 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.236559    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 15:44:06.236761 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.236562    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 15:44:06.236761 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.236564    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 15:44:06.236761 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.236567    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 15:44:06.237256 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.236570    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 15:44:06.237256 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.236572    2566 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 15:44:06.237256 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.236575    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 15:44:06.237256 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.236577    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 15:44:06.237256 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.236580    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 15:44:06.237256 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.236590    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 15:44:06.237256 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.236593    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 15:44:06.237256 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.236595    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 15:44:06.237256 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.236598    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 15:44:06.237256 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.236600    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 15:44:06.237256 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.236603    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 15:44:06.237256 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.236606    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 15:44:06.237256 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.236608    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 15:44:06.237256 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.236612    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 15:44:06.237256 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.236615    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 15:44:06.237256 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.236618    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 15:44:06.237256 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.236620    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 15:44:06.237256 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.236623    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 15:44:06.237256 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.236626    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 15:44:06.237256 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.236628    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 15:44:06.237780 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.236631    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 15:44:06.237780 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.236633    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 15:44:06.237780 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.236636    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 15:44:06.237780 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.236638    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 15:44:06.237780 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.236641    2566 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 15:44:06.237780 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.236643    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 15:44:06.237780 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.236646    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 15:44:06.237780 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.236649    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 15:44:06.237780 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.236651    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 15:44:06.237780 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.236654    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 15:44:06.237780 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.236657    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 15:44:06.237780 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.236659    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 15:44:06.237780 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.236663    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 15:44:06.237780 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.236666    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 15:44:06.237780 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.236668    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 15:44:06.237780 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.236671    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 15:44:06.237780 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.236673    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 15:44:06.237780 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.236676    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 15:44:06.237780 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.236678    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 15:44:06.237780 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.236681    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 15:44:06.238268 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.236683    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 15:44:06.238268 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.236686    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 15:44:06.238268 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.236688    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 15:44:06.238268 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.236693    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 15:44:06.238268 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.236697    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 15:44:06.238268 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.236703    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 15:44:06.238268 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.236707    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 15:44:06.238268 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.236710    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 15:44:06.238268 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.236713    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 15:44:06.238268 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.236716    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 15:44:06.238268 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.236718    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 15:44:06.238268 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.236721    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 15:44:06.238268 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.236725    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 15:44:06.238268 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.236728    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 15:44:06.238268 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.236730    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 15:44:06.238268 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.236733    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 15:44:06.238268 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.236735    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 15:44:06.238268 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.236739    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 15:44:06.238268 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.236742    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 15:44:06.238758 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.236745    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 15:44:06.238758 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.236747    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 15:44:06.238758 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.236751    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 15:44:06.238758 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.236755    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 15:44:06.238758 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.236758    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 15:44:06.238758 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.236761    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 15:44:06.238758 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.236764    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 15:44:06.238758 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.236767    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 15:44:06.238758 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.236770    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 15:44:06.238758 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.236773    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 15:44:06.238758 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.236776    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 15:44:06.238758 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.236779    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 15:44:06.238758 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.236781    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 15:44:06.238758 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.236784    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 15:44:06.238758 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.237508    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 15:44:06.238758 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.237515    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 15:44:06.238758 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.237518    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 15:44:06.238758 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.237521    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 15:44:06.238758 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.237525    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 15:44:06.238758 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.237528    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 15:44:06.239302 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.237532    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 15:44:06.239302 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.237536    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 15:44:06.239302 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.237539    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 15:44:06.239302 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.237542    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 15:44:06.239302 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.237545    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 15:44:06.239302 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.237548    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 15:44:06.239302 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.237550    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 15:44:06.239302 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.237553    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 15:44:06.239302 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.237555    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 15:44:06.239302 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.237558    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 15:44:06.239302 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.237560    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 15:44:06.239302 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.237563    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 15:44:06.239302 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.237566    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 15:44:06.239302 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.237569    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 15:44:06.239302 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.237571    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 15:44:06.239302 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.237574    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 15:44:06.239302 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.237577    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 15:44:06.239302 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.237579    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 15:44:06.239302 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.237582    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 15:44:06.239796 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.237584    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 15:44:06.239796 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.237587    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 15:44:06.239796 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.237590    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 15:44:06.239796 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.237594    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 15:44:06.239796 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.237597    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 15:44:06.239796 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.237600    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 15:44:06.239796 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.237603    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 15:44:06.239796 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.237605    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 15:44:06.239796 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.237608    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 15:44:06.239796 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.237611    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 15:44:06.239796 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.237613    2566 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 15:44:06.239796 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.237617    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 15:44:06.239796 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.237620    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 15:44:06.239796 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.237623    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 15:44:06.239796 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.237634    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 15:44:06.239796 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.237637    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 15:44:06.239796 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.237640    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 15:44:06.239796 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.237643    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 15:44:06.239796 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.237646    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 15:44:06.239796 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.237648    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 15:44:06.240297 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.237651    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 15:44:06.240297 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.237653    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 15:44:06.240297 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.237656    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 15:44:06.240297 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.237659    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 15:44:06.240297 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.237662    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 15:44:06.240297 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.237665    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 15:44:06.240297 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.237667    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 15:44:06.240297 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.237670    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 15:44:06.240297 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.237673    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 15:44:06.240297 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.237675    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 15:44:06.240297 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.237679    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 15:44:06.240297 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.237681    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 15:44:06.240297 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.237684    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 15:44:06.240297 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.237686    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 15:44:06.240297 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.237689    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 15:44:06.240297 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.237691    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 15:44:06.240297 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.237694    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 15:44:06.240297 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.237697    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 15:44:06.240297 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.237699    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 15:44:06.240297 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.237702    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 15:44:06.240778 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.237704    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 15:44:06.240778 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.237707    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 15:44:06.240778 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.237709    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 15:44:06.240778 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.237713    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 15:44:06.240778 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.237716    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 15:44:06.240778 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.237719    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 15:44:06.240778 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.237721    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 15:44:06.240778 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.237724    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 15:44:06.240778 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.237726    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 15:44:06.240778 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.237729    2566 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 15:44:06.240778 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.237731    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 15:44:06.240778 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.237734    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 15:44:06.240778 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.237736    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 15:44:06.240778 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.237739    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 15:44:06.240778 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.237741    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 15:44:06.240778 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.237744    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 15:44:06.240778 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.237746    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 15:44:06.240778 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.237749    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 15:44:06.240778 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.237752    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 15:44:06.240778 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.237754    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 15:44:06.241283 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.237757    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 15:44:06.241283 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.237759    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 15:44:06.241283 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.237761    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 15:44:06.241283 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.237766    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 15:44:06.241283 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.237768    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 15:44:06.241283 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.237771    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 15:44:06.241283 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.237774    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 15:44:06.241283 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.237776    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 15:44:06.241283 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.237779    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 15:44:06.241283 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.237782    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 15:44:06.241283 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.237785    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 15:44:06.241283 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.237787    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 15:44:06.241283 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.237791    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 15:44:06.241283 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.237795    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 15:44:06.241283 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.238198    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 15:44:06.241283 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.238203    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 15:44:06.241283 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.238206    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 15:44:06.241283 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.238209    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 15:44:06.241283 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.238211    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 15:44:06.241283 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.238214    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 15:44:06.241790 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.238217    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 15:44:06.241790 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.238219    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 15:44:06.241790 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.238222    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 15:44:06.241790 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.238225    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 15:44:06.241790 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.238227    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 15:44:06.241790 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.238230    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 15:44:06.241790 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.238233    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 15:44:06.241790 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.238236    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 15:44:06.241790 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.238238    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 15:44:06.241790 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.238241    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 15:44:06.241790 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.238243    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 15:44:06.241790 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.238248    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 15:44:06.241790 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.238252    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 15:44:06.241790 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.238255    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 15:44:06.241790 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.238257    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 15:44:06.241790 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.238260    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 15:44:06.241790 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.238264    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 15:44:06.241790 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.238266    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 15:44:06.241790 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.238269    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 15:44:06.242274 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.238274    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 15:44:06.242274 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.238277    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 15:44:06.242274 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.238280    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 15:44:06.242274 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.238283    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 15:44:06.242274 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.238286    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 15:44:06.242274 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.238289    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 15:44:06.242274 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.238292    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 15:44:06.242274 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.238296    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 15:44:06.242274 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.238299    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 15:44:06.242274 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.238302    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 15:44:06.242274 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.238306    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 15:44:06.242274 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.238309    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 15:44:06.242274 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.238312    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 15:44:06.242274 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.238315    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 15:44:06.242274 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.238317    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 15:44:06.242274 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.238320    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 15:44:06.242274 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.238322    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 15:44:06.242274 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.238325    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 15:44:06.242274 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.238327    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 15:44:06.242742 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.238330    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 15:44:06.242742 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.238332    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 15:44:06.242742 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.238335    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 15:44:06.242742 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.238337    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 15:44:06.242742 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.238340    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 15:44:06.242742 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.238343    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 15:44:06.242742 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.238346    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 15:44:06.242742 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.238349    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 15:44:06.242742 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.238351    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 15:44:06.242742 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.238354    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 15:44:06.242742 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.238356    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 15:44:06.242742 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.238359    2566 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 15:44:06.242742 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.238361    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 15:44:06.242742 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.238366    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 15:44:06.242742 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.238369    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 15:44:06.242742 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.238371    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 15:44:06.242742 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.238373    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 15:44:06.242742 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.238376    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 15:44:06.242742 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.238379    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 15:44:06.242742 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.238381    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 15:44:06.243245 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.238384    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 15:44:06.243245 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.238387    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 15:44:06.243245 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.238390    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 15:44:06.243245 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.238392    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 15:44:06.243245 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.238395    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 15:44:06.243245 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.238399    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 15:44:06.243245 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.238401    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 15:44:06.243245 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.238404    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 15:44:06.243245 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.238406    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 15:44:06.243245 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.238409    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 15:44:06.243245 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.238412    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 15:44:06.243245 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.238414    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 15:44:06.243245 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.238417    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 15:44:06.243245 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.238419    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 15:44:06.243245 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.238422    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 15:44:06.243245 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.238425    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 15:44:06.243245 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.238427    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 15:44:06.243245 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.238430    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 15:44:06.243245 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.238432    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 15:44:06.243245 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.238435    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 15:44:06.243727 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.238438    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 15:44:06.243727 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.238440    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 15:44:06.243727 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.238443    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 15:44:06.243727 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.238445    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 15:44:06.243727 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.238448    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 15:44:06.243727 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.238452    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 15:44:06.243727 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.238455    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 15:44:06.243727 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.238457    2566 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 15:44:06.243727 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.238460    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 15:44:06.243727 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.238463    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 15:44:06.243727 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.238465    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 15:44:06.243727 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.238468    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 15:44:06.243727 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.238470    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 15:44:06.243727 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.238474    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 15:44:06.243727 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.238477    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 15:44:06.243727 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238551    2566 flags.go:64] FLAG: --address="0.0.0.0"
Jul 07 15:44:06.243727 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238562    2566 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 07 15:44:06.243727 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238569    2566 flags.go:64] FLAG: --anonymous-auth="true"
Jul 07 15:44:06.243727 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238574    2566 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 07 15:44:06.243727 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238578    2566 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 07 15:44:06.243727 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238581    2566 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 07 15:44:06.244263 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238585    2566 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 07 15:44:06.244263 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238590    2566 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 07 15:44:06.244263 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238593    2566 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 07 15:44:06.244263 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238596    2566 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 07 15:44:06.244263 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238599    2566 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 07 15:44:06.244263 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238603    2566 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 07 15:44:06.244263 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238606    2566 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 07 15:44:06.244263 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238609    2566 flags.go:64] FLAG: --cgroup-root=""
Jul 07 15:44:06.244263 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238612    2566 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 07 15:44:06.244263 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238615    2566 flags.go:64] FLAG: --client-ca-file=""
Jul 07 15:44:06.244263 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238618    2566 flags.go:64] FLAG: --cloud-provider="external"
Jul 07 15:44:06.244263 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238622    2566 flags.go:64] FLAG: --cluster-dns="[]"
Jul 07 15:44:06.244263 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238626    2566 flags.go:64] FLAG: --cluster-domain=""
Jul 07 15:44:06.244263 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238628    2566 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 07 15:44:06.244263 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238631    2566 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 07 15:44:06.244263 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238634    2566 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 07 15:44:06.244263 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238639    2566 flags.go:64] FLAG: --container-log-max-files="5"
Jul 07 15:44:06.244263 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238643    2566 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 07 15:44:06.244263 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238646    2566 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 07 15:44:06.244263 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238649    2566 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 07 15:44:06.244263 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238653    2566 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 07 15:44:06.244263 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238656    2566 flags.go:64] FLAG: --contention-profiling="false"
Jul 07 15:44:06.244263 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238659    2566 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 07 15:44:06.244263 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238662    2566 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 07 15:44:06.244832 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238666    2566 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 07 15:44:06.244832 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238670    2566 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 07 15:44:06.244832 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238674    2566 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 07 15:44:06.244832 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238677    2566 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 07 15:44:06.244832 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238680    2566 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 07 15:44:06.244832 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238683    2566 flags.go:64] FLAG: --enable-load-reader="false"
Jul 07 15:44:06.244832 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238686    2566 flags.go:64] FLAG: --enable-server="true"
Jul 07 15:44:06.244832 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238689    2566 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 07 15:44:06.244832 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238693    2566 flags.go:64] FLAG: --event-burst="100"
Jul 07 15:44:06.244832 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238696    2566 flags.go:64] FLAG: --event-qps="50"
Jul 07 15:44:06.244832 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238699    2566 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 07 15:44:06.244832 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238702    2566 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 07 15:44:06.244832 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238705    2566 flags.go:64] FLAG: --eviction-hard=""
Jul 07 15:44:06.244832 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238709    2566 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 07 15:44:06.244832 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238712    2566 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 07 15:44:06.244832 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238715    2566 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 07 15:44:06.244832 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238717    2566 flags.go:64] FLAG: --eviction-soft=""
Jul 07 15:44:06.244832 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238720    2566 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 07 15:44:06.244832 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238723    2566 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 07 15:44:06.244832 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238726    2566 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 07 15:44:06.244832 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238729    2566 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 07 15:44:06.244832 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238732    2566 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 07 15:44:06.244832 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238735    2566 flags.go:64] FLAG: --fail-swap-on="true"
Jul 07 15:44:06.244832 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238738    2566 flags.go:64] FLAG: --feature-gates=""
Jul 07 15:44:06.244832 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238743    2566 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 07 15:44:06.245564 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238746    2566 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 07 15:44:06.245564 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238749    2566 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 07 15:44:06.245564 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238752    2566 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 07 15:44:06.245564 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238755    2566 flags.go:64] FLAG: --healthz-port="10248"
Jul 07 15:44:06.245564 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238758    2566 flags.go:64] FLAG: --help="false"
Jul 07 15:44:06.245564 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238761    2566 flags.go:64] FLAG: --hostname-override="ip-10-0-138-101.ec2.internal"
Jul 07 15:44:06.245564 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238764    2566 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 07 15:44:06.245564 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238769    2566 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 07 15:44:06.245564 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238772    2566 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 07 15:44:06.245564 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238775    2566 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 07 15:44:06.245564 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238779    2566 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 07 15:44:06.245564 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238781    2566 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 07 15:44:06.245564 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238784    2566 flags.go:64] FLAG: --image-service-endpoint=""
Jul 07 15:44:06.245564 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238787    2566 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 07 15:44:06.245564 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238790    2566 flags.go:64] FLAG: --kube-api-burst="100"
Jul 07 15:44:06.245564 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238792    2566 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 07 15:44:06.245564 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238796    2566 flags.go:64] FLAG: --kube-api-qps="50"
Jul 07 15:44:06.245564 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238798    2566 flags.go:64] FLAG: --kube-reserved=""
Jul 07 15:44:06.245564 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238801    2566 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 07 15:44:06.245564 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238804    2566 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 07 15:44:06.245564 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238807    2566 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 07 15:44:06.245564 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238810    2566 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 07 15:44:06.245564 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238813    2566 flags.go:64] FLAG: --lock-file=""
Jul 07 15:44:06.245564 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238816    2566 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 07 15:44:06.246222 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238819    2566 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 07 15:44:06.246222 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238822    2566 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 07 15:44:06.246222 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238827    2566 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 07 15:44:06.246222 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238831    2566 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 07 15:44:06.246222 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238834    2566 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 07 15:44:06.246222 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238837    2566 flags.go:64] FLAG: --logging-format="text"
Jul 07 15:44:06.246222 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238840    2566 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 07 15:44:06.246222 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238844    2566 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 07 15:44:06.246222 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238846    2566 flags.go:64] FLAG: --manifest-url=""
Jul 07 15:44:06.246222 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238849    2566 flags.go:64] FLAG: --manifest-url-header=""
Jul 07 15:44:06.246222 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238856    2566 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 07 15:44:06.246222 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238859    2566 flags.go:64] FLAG: --max-open-files="1000000"
Jul 07 15:44:06.246222 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238863    2566 flags.go:64] FLAG: --max-pods="110"
Jul 07 15:44:06.246222 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238866    2566 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 07 15:44:06.246222 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238869    2566 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 07 15:44:06.246222 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238873    2566 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 07 15:44:06.246222 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238876    2566 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 07 15:44:06.246222 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238880    2566 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 07 15:44:06.246222 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238895    2566 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 07 15:44:06.246222 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238898    2566 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 07 15:44:06.246222 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238906    2566 flags.go:64] FLAG: --node-status-max-images="50"
Jul 07 15:44:06.246222 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238909    2566 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 07 15:44:06.246222 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238912    2566 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 07 15:44:06.246222 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238914    2566 flags.go:64] FLAG: --pod-cidr=""
Jul 07 15:44:06.246851 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238917    2566 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5"
Jul 07 15:44:06.246851 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238923    2566 flags.go:64] FLAG: --pod-manifest-path=""
Jul 07 15:44:06.246851 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238925    2566 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 07 15:44:06.246851 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238929    2566 flags.go:64] FLAG: --pods-per-core="0"
Jul 07 15:44:06.246851 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238931    2566 flags.go:64] FLAG: --port="10250"
Jul 07 15:44:06.246851 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238934    2566 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 07 15:44:06.246851 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238937    2566 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0b883e28729c7b217"
Jul 07 15:44:06.246851 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238941    2566 flags.go:64] FLAG: --qos-reserved=""
Jul 07 15:44:06.246851 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238944    2566 flags.go:64] FLAG: --read-only-port="10255"
Jul 07 15:44:06.246851 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238947    2566 flags.go:64] FLAG: --register-node="true"
Jul 07 15:44:06.246851 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238950    2566 flags.go:64] FLAG: --register-with-taints=""
Jul 07 15:44:06.246851 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238953    2566 flags.go:64] FLAG: --registry-burst="10"
Jul 07 15:44:06.246851 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238957    2566 flags.go:64] FLAG: --registry-qps="5"
Jul 07 15:44:06.246851 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238960    2566 flags.go:64] FLAG: --reserved-cpus=""
Jul 07 15:44:06.246851 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238962    2566 flags.go:64] FLAG: --reserved-memory=""
Jul 07 15:44:06.246851 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238966    2566 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 07 15:44:06.246851 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238969    2566 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 07 15:44:06.246851 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238972    2566 flags.go:64] FLAG: --rotate-certificates="false"
Jul 07 15:44:06.246851 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238975    2566 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 07 15:44:06.246851 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238978    2566 flags.go:64] FLAG: --runonce="false"
Jul 07 15:44:06.246851 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238981    2566 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 07 15:44:06.246851 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238984    2566 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 07 15:44:06.246851 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238987    2566 flags.go:64] FLAG: --seccomp-default="false"
Jul 07 15:44:06.246851 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238990    2566 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 07 15:44:06.246851 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238994    2566 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 07 15:44:06.247485 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.238997    2566 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 07 15:44:06.247485 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.239000    2566 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 07 15:44:06.247485 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.239003    2566 flags.go:64] FLAG: --storage-driver-password="root"
Jul 07 15:44:06.247485 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.239006    2566 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 07 15:44:06.247485 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.239009    2566 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 07 15:44:06.247485 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.239012    2566 flags.go:64] FLAG: --storage-driver-user="root"
Jul 07 15:44:06.247485 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.239015    2566 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 07 15:44:06.247485 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.239018    2566 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 07 15:44:06.247485 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.239021    2566 flags.go:64] FLAG: --system-cgroups=""
Jul 07 15:44:06.247485 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.239024    2566 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 07 15:44:06.247485 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.239029    2566 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 07 15:44:06.247485 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.239032    2566 flags.go:64] FLAG: --tls-cert-file=""
Jul 07 15:44:06.247485 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.239034    2566 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 07 15:44:06.247485 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.239038    2566 flags.go:64] FLAG: --tls-min-version=""
Jul 07 15:44:06.247485 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.239041    2566 flags.go:64] FLAG: --tls-private-key-file=""
Jul 07 15:44:06.247485 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.239044    2566 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 07 15:44:06.247485 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.239047    2566 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 07 15:44:06.247485 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.239049    2566 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 07 15:44:06.247485 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.239052    2566 flags.go:64] FLAG: --v="2"
Jul 07 15:44:06.247485 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.239056    2566 flags.go:64] FLAG: --version="false"
Jul 07 15:44:06.247485 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.239061    2566 flags.go:64] FLAG: --vmodule=""
Jul 07 15:44:06.247485 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.239065    2566 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 07 15:44:06.247485 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.239069    2566 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 07 15:44:06.247485 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.239182    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 15:44:06.247485 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.239186    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 15:44:06.248114 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.239189    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 15:44:06.248114 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.239191    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 15:44:06.248114 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.239195    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 15:44:06.248114 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.239197    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 15:44:06.248114 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.239200    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 15:44:06.248114 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.239203    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 15:44:06.248114 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.239208    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 15:44:06.248114 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.239210    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 15:44:06.248114 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.239213    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 15:44:06.248114 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.239216    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 15:44:06.248114 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.239218    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 15:44:06.248114 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.239221    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 15:44:06.248114 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.239224    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 15:44:06.248114 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.239226    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 15:44:06.248114 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.239229    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 15:44:06.248114 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.239232    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 15:44:06.248114 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.239235    2566 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 15:44:06.248114 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.239237    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 15:44:06.248114 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.239240    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 15:44:06.248114 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.239242    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 15:44:06.248597 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.239245    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 15:44:06.248597 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.239247    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 15:44:06.248597 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.239250    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 15:44:06.248597 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.239252    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 15:44:06.248597 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.239255    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 15:44:06.248597 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.239258    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 15:44:06.248597 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.239260    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 15:44:06.248597 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.239263    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 15:44:06.248597 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.239266    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 15:44:06.248597 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.239269    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 15:44:06.248597 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.239271    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 15:44:06.248597 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.239274    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 15:44:06.248597 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.239278    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 15:44:06.248597 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.239281    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 15:44:06.248597 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.239284    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 15:44:06.248597 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.239287    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 15:44:06.248597 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.239290    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 15:44:06.248597 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.239292    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 15:44:06.248597 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.239298    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 15:44:06.249116 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.239301    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 15:44:06.249116 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.239304    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 15:44:06.249116 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.239307    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 15:44:06.249116 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.239310    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 15:44:06.249116 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.239313    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 15:44:06.249116 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.239315    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 15:44:06.249116 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.239318    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 15:44:06.249116 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.239321    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 15:44:06.249116 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.239323    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 15:44:06.249116 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.239326    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 15:44:06.249116 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.239328    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 15:44:06.249116 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.239331    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 15:44:06.249116 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.239334    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 15:44:06.249116 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.239336    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 15:44:06.249116 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.239339    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 15:44:06.249116 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.239341    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 15:44:06.249116 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.239344    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 15:44:06.249116 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.239347    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 15:44:06.249116 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.239349    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 15:44:06.249116 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.239352    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 15:44:06.249596 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.239354    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 15:44:06.249596 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.239357    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 15:44:06.249596 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.239361    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 15:44:06.249596 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.239364    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 15:44:06.249596 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.239367    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 15:44:06.249596 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.239371    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 15:44:06.249596 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.239374    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 15:44:06.249596 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.239376    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 15:44:06.249596 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.239379    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 15:44:06.249596 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.239382    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 15:44:06.249596 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.239384    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 15:44:06.249596 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.239388    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 15:44:06.249596 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.239390    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 15:44:06.249596 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.239393    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 15:44:06.249596 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.239396    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 15:44:06.249596 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.239398    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 15:44:06.249596 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.239401    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 15:44:06.249596 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.239404    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 15:44:06.249596 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.239406    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 15:44:06.250083 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.239409    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 15:44:06.250083 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.239411    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 15:44:06.250083 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.239414    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 15:44:06.250083 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.239417    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 15:44:06.250083 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.239419    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 15:44:06.250083 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.239422    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 15:44:06.250083 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.239424    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 15:44:06.250083 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.239427    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 15:44:06.250083 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.239430    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 15:44:06.250083 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.239432    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 15:44:06.250083 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.239435    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 15:44:06.250083 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.239437    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 15:44:06.250083 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.239440    2566 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 15:44:06.250083 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.239443    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 15:44:06.250083 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.239445    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 15:44:06.250083 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.239449    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 15:44:06.250083 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.239452    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 15:44:06.250083 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.239455    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 15:44:06.250083 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.239457    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 15:44:06.250546 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.240202    2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 07 15:44:06.250546 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.247319    2566 server.go:529] "Kubelet version" kubeletVersion="v1.34.8"
Jul 07 15:44:06.250546 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.247337    2566 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 07 15:44:06.250546 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.247352    2566 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 07 15:44:06.250546 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.247358    2566 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 07 15:44:06.250546 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.247413    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 15:44:06.250546 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.247418    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 15:44:06.250546 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.247421    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 15:44:06.250546 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.247424    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 15:44:06.250546 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.247428    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 15:44:06.250546 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.247431    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 15:44:06.250546 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.247433    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 15:44:06.250546 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.247436    2566 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 15:44:06.250546 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.247439    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 15:44:06.250546 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.247441    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 15:44:06.250960 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.247444    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 15:44:06.250960 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.247447    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 15:44:06.250960 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.247449    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 15:44:06.250960 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.247452    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 15:44:06.250960 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.247455    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 15:44:06.250960 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.247458    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 15:44:06.250960 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.247460    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 15:44:06.250960 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.247463    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 15:44:06.250960 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.247466    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 15:44:06.250960 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.247469    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 15:44:06.250960 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.247471    2566 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 15:44:06.250960 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.247474    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 15:44:06.250960 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.247477    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 15:44:06.250960 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.247479    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 15:44:06.250960 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.247482    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 15:44:06.250960 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.247484    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 15:44:06.250960 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.247487    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 15:44:06.250960 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.247490    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 15:44:06.250960 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.247492    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 15:44:06.250960 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.247847    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 15:44:06.251474 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.247851    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 15:44:06.251474 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.247854    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 15:44:06.251474 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.247857    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 15:44:06.251474 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.247860    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 15:44:06.251474 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.247863    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 15:44:06.251474 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.247865    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 15:44:06.251474 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.247868    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 15:44:06.251474 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.247870    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 15:44:06.251474 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.247873    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 15:44:06.251474 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.247876    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 15:44:06.251474 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.247878    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 15:44:06.251474 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.247898    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 15:44:06.251474 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.247901    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 15:44:06.251474 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.247904    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 15:44:06.251474 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.247907    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 15:44:06.251474 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.247910    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 15:44:06.251474 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.247913    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 15:44:06.251474 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.247916    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 15:44:06.251474 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.247918    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 15:44:06.251474 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.247921    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 15:44:06.251988 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.247924    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 15:44:06.251988 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.247926    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 15:44:06.251988 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.247929    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 15:44:06.251988 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.247931    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 15:44:06.251988 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.247934    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 15:44:06.251988 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.247936    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 15:44:06.251988 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.247939    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 15:44:06.251988 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.247941    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 15:44:06.251988 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.247944    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 15:44:06.251988 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.247947    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 15:44:06.251988 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.247949    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 15:44:06.251988 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.247952    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 15:44:06.251988 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.247956    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 15:44:06.251988 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.247958    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 15:44:06.251988 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.247961    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 15:44:06.251988 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.247964    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 15:44:06.251988 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.247966    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 15:44:06.251988 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.247970    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 15:44:06.251988 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.247974    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 15:44:06.251988 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.247976    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 15:44:06.252462 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.247979    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 15:44:06.252462 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.247983    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 15:44:06.252462 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.247987    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 15:44:06.252462 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.247990    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 15:44:06.252462 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.247993    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 15:44:06.252462 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.247996    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 15:44:06.252462 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.247999    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 15:44:06.252462 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248001    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 15:44:06.252462 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248004    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 15:44:06.252462 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248006    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 15:44:06.252462 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248009    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 15:44:06.252462 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248012    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 15:44:06.252462 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248015    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 15:44:06.252462 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248017    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 15:44:06.252462 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248020    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 15:44:06.252462 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248022    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 15:44:06.252462 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248025    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 15:44:06.252462 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248028    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 15:44:06.252462 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248030    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 15:44:06.253004 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248034    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 15:44:06.253004 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248037    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 15:44:06.253004 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248039    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 15:44:06.253004 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248042    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 15:44:06.253004 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248044    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 15:44:06.253004 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248048    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 15:44:06.253004 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248052    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 15:44:06.253004 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248055    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 15:44:06.253004 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248059    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 15:44:06.253004 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248062    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 15:44:06.253004 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.248067    2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 07 15:44:06.253004 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248180    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 15:44:06.253004 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248184    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 15:44:06.253004 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248187    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 15:44:06.253353 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248190    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 15:44:06.253353 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248192    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 15:44:06.253353 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248195    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 15:44:06.253353 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248197    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 15:44:06.253353 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248200    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 15:44:06.253353 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248202    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 15:44:06.253353 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248205    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 15:44:06.253353 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248207    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 15:44:06.253353 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248210    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 15:44:06.253353 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248212    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 15:44:06.253353 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248215    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 15:44:06.253353 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248217    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 15:44:06.253353 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248220    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 15:44:06.253353 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248223    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 15:44:06.253353 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248225    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 15:44:06.253353 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248228    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 15:44:06.253353 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248230    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 15:44:06.253353 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248233    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 15:44:06.253353 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248235    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 15:44:06.253353 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248238    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 15:44:06.253857 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248241    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 15:44:06.253857 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248243    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 15:44:06.253857 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248246    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 15:44:06.253857 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248248    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 15:44:06.253857 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248251    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 15:44:06.253857 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248254    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 15:44:06.253857 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248257    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 15:44:06.253857 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248260    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 15:44:06.253857 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248264    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 15:44:06.253857 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248267    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 15:44:06.253857 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248270    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 15:44:06.253857 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248273    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 15:44:06.253857 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248276    2566 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 15:44:06.253857 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248278    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 15:44:06.253857 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248281    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 15:44:06.253857 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248284    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 15:44:06.253857 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248286    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 15:44:06.253857 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248289    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 15:44:06.253857 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248292    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 15:44:06.254339 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248296    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 15:44:06.254339 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248299    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 15:44:06.254339 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248301    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 15:44:06.254339 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248304    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 15:44:06.254339 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248306    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 15:44:06.254339 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248309    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 15:44:06.254339 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248311    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 15:44:06.254339 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248314    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 15:44:06.254339 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248316    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 15:44:06.254339 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248319    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 15:44:06.254339 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248322    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 15:44:06.254339 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248324    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 15:44:06.254339 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248327    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 15:44:06.254339 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248330    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 15:44:06.254339 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248332    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 15:44:06.254339 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248335    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 15:44:06.254339 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248338    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 15:44:06.254339 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248340    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 15:44:06.254339 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248344    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 15:44:06.254786 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248347    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 15:44:06.254786 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248350    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 15:44:06.254786 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248353    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 15:44:06.254786 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248356    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 15:44:06.254786 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248360    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 15:44:06.254786 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248362    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 15:44:06.254786 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248365    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 15:44:06.254786 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248368    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 15:44:06.254786 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248370    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 15:44:06.254786 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248372    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 15:44:06.254786 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248375    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 15:44:06.254786 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248378    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 15:44:06.254786 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248380    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 15:44:06.254786 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248383    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 15:44:06.254786 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248385    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 15:44:06.254786 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248388    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 15:44:06.254786 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248390    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 15:44:06.254786 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248393    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 15:44:06.254786 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248395    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 15:44:06.254786 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248398    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 15:44:06.255293 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248400    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 15:44:06.255293 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248403    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 15:44:06.255293 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248405    2566 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 15:44:06.255293 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248408    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 15:44:06.255293 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248411    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 15:44:06.255293 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248413    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 15:44:06.255293 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248416    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 15:44:06.255293 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248420    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 15:44:06.255293 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248423    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 15:44:06.255293 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248425    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 15:44:06.255293 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248428    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 15:44:06.255293 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248430    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 15:44:06.255293 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248433    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 15:44:06.255293 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248436    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 15:44:06.255293 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248438    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 15:44:06.255293 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248441    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 15:44:06.255293 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248443    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 15:44:06.255293 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:06.248446    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 15:44:06.255735 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.248451    2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 07 15:44:06.255735 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.248553    2566 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 07 15:44:06.255735 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.250939    2566 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 07 15:44:06.255735 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.252670    2566 server.go:1019] "Starting client certificate rotation"
Jul 07 15:44:06.255735 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.252763    2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 07 15:44:06.255735 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.252801    2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 07 15:44:06.256155 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.256141    2566 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 07 15:44:06.259532 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.259508    2566 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 07 15:44:06.276996 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.276971    2566 log.go:25] "Validated CRI v1 runtime API"
Jul 07 15:44:06.279458 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.279443    2566 log.go:25] "Validated CRI v1 image API"
Jul 07 15:44:06.280327 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.280312    2566 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 07 15:44:06.284212 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.284191    2566 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 07 15:44:06.284642 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.284615    2566 fs.go:135] Filesystem UUIDs: map[56b007dc-9e7c-48a3-8b43-6ada84f793f1:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 9fc70a85-baf3-4a61-9b0a-1db72c4fb54b:/dev/nvme0n1p4]
Jul 07 15:44:06.284687 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.284643    2566 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 07 15:44:06.290678 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.290573    2566 manager.go:216] Machine: {Timestamp:2026-07-07 15:44:06.289282064 +0000 UTC m=+0.377707725 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099843 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22872e2f0f2deed67f902eb6631dd7 SystemUUID:ec22872e-2f0f-2dee-d67f-902eb6631dd7 BootID:889f7d3e-e16a-4016-9da6-016284b89115 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:6a:55:2e:30:c5 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:6a:55:2e:30:c5 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:22:dc:d8:9e:fc:24 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 07 15:44:06.290678 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.290673    2566 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 07 15:44:06.290778 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.290749    2566 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 07 15:44:06.292020 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.291995    2566 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 07 15:44:06.292172 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.292025    2566 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-101.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 07 15:44:06.293655 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.293644    2566 topology_manager.go:138] "Creating topology manager with none policy"
Jul 07 15:44:06.293693 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.293658    2566 container_manager_linux.go:309] "Creating device plugin manager"
Jul 07 15:44:06.293693 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.293665    2566 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 07 15:44:06.294481 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.294471    2566 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 07 15:44:06.296341 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.296332    2566 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 07 15:44:06.300946 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.300934    2566 state_mem.go:36] "Initialized new in-memory state store"
Jul 07 15:44:06.301101 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.301091    2566 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 07 15:44:06.301133 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.301121    2566 kubelet.go:486] "Attempting to sync node with API server"
Jul 07 15:44:06.301160 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.301135    2566 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 07 15:44:06.301160 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.301147    2566 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 07 15:44:06.301160 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.301156    2566 kubelet.go:398] "Adding apiserver pod source"
Jul 07 15:44:06.301270 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.301165    2566 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 07 15:44:06.301514 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.301502    2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 07 15:44:06.301567 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.301521    2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 07 15:44:06.303971 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.303956    2566 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1"
Jul 07 15:44:06.306794 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.306780    2566 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 07 15:44:06.306850 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.306811    2566 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 07 15:44:06.308164 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.308152    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 07 15:44:06.308207 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.308170    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 07 15:44:06.308207 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.308176    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 07 15:44:06.308207 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.308181    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 07 15:44:06.308207 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.308189    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 07 15:44:06.308207 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.308194    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 07 15:44:06.308207 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.308203    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 07 15:44:06.308207 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.308208    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 07 15:44:06.308393 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.308215    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 07 15:44:06.308393 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.308220    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 07 15:44:06.308393 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.308228    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 07 15:44:06.308393 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.308233    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 07 15:44:06.309107 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.309097    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 07 15:44:06.309107 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.309105    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 07 15:44:06.312385 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.312367    2566 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-101.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 07 15:44:06.312479 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.312434    2566 server.go:1268] "Started kubelet"
Jul 07 15:44:06.312479 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:06.312434    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 07 15:44:06.312479 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:06.312434    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-101.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 07 15:44:06.312603 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.312492    2566 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 07 15:44:06.312603 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.312506    2566 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 07 15:44:06.312603 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.312597    2566 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 07 15:44:06.313318 ip-10-0-138-101 systemd[1]: Started Kubernetes Kubelet.
Jul 07 15:44:06.317254 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.317227    2566 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 07 15:44:06.318919 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:06.318896    2566 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 07 15:44:06.319165 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.319149    2566 server.go:310] "Adding debug handlers to kubelet server"
Jul 07 15:44:06.323968 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:06.322837    2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-101.ec2.internal.18c00c5d120338bc  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-101.ec2.internal,UID:ip-10-0-138-101.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-101.ec2.internal,},FirstTimestamp:2026-07-07 15:44:06.312409276 +0000 UTC m=+0.400834984,LastTimestamp:2026-07-07 15:44:06.312409276 +0000 UTC m=+0.400834984,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-101.ec2.internal,}"
Jul 07 15:44:06.326601 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.326579    2566 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 07 15:44:06.326601 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.326597    2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 07 15:44:06.327390 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.327363    2566 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 07 15:44:06.327846 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.327827    2566 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 07 15:44:06.327936 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.327849    2566 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 07 15:44:06.328024 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.328011    2566 reconstruct.go:97] "Volume reconstruction finished"
Jul 07 15:44:06.328024 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.328024    2566 reconciler.go:29] "Reconciler: start to sync state"
Jul 07 15:44:06.328172 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:06.328152    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-101.ec2.internal\" not found"
Jul 07 15:44:06.328252 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.328239    2566 factory.go:55] Registering systemd factory
Jul 07 15:44:06.328301 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.328257    2566 factory.go:223] Registration of the systemd container factory successfully
Jul 07 15:44:06.328481 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.328462    2566 factory.go:153] Registering CRI-O factory
Jul 07 15:44:06.328481 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.328481    2566 factory.go:223] Registration of the crio container factory successfully
Jul 07 15:44:06.328629 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.328528    2566 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 07 15:44:06.328629 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.328549    2566 factory.go:103] Registering Raw factory
Jul 07 15:44:06.328629 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.328562    2566 manager.go:1196] Started watching for new ooms in manager
Jul 07 15:44:06.329189 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.329175    2566 manager.go:349] Starting recovery of all containers
Jul 07 15:44:06.329944 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:06.329880    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 07 15:44:06.330015 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:06.329944    2566 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-101.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 07 15:44:06.339817 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.339801    2566 manager.go:354] Recovery completed
Jul 07 15:44:06.343705 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.343693    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 15:44:06.345486 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.345462    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-101.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 15:44:06.345596 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.345502    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-101.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 15:44:06.345596 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.345517    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-101.ec2.internal" event="NodeHasSufficientPID"
Jul 07 15:44:06.346010 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.345990    2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-xq7fg"
Jul 07 15:44:06.346127 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.346116    2566 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 07 15:44:06.346127 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.346126    2566 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 07 15:44:06.346210 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.346134    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 15:44:06.346210 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.346141    2566 state_mem.go:36] "Initialized new in-memory state store"
Jul 07 15:44:06.346819 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.346806    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-101.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 15:44:06.346870 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.346829    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-101.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 15:44:06.346870 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.346839    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-101.ec2.internal" event="NodeHasSufficientPID"
Jul 07 15:44:06.347379 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:06.347297    2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-101.ec2.internal.18c00c5d13fbeb11  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-101.ec2.internal,UID:ip-10-0-138-101.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-101.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-101.ec2.internal,},FirstTimestamp:2026-07-07 15:44:06.345485073 +0000 UTC m=+0.433910944,LastTimestamp:2026-07-07 15:44:06.345485073 +0000 UTC m=+0.433910944,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-101.ec2.internal,}"
Jul 07 15:44:06.348352 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.348339    2566 policy_none.go:49] "None policy: Start"
Jul 07 15:44:06.348404 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.348357    2566 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 07 15:44:06.348404 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.348368    2566 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 07 15:44:06.350224 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.350213    2566 policy_none.go:47] "Start"
Jul 07 15:44:06.356152 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:06.355977    2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-101.ec2.internal.18c00c5d13fc48ea  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-101.ec2.internal,UID:ip-10-0-138-101.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-138-101.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-138-101.ec2.internal,},FirstTimestamp:2026-07-07 15:44:06.345509098 +0000 UTC m=+0.433934760,LastTimestamp:2026-07-07 15:44:06.345509098 +0000 UTC m=+0.433934760,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-101.ec2.internal,}"
Jul 07 15:44:06.356559 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.356545    2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-xq7fg"
Jul 07 15:44:06.360063 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:06.359966    2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-101.ec2.internal.18c00c5d13fc7f5a  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-101.ec2.internal,UID:ip-10-0-138-101.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-138-101.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-138-101.ec2.internal,},FirstTimestamp:2026-07-07 15:44:06.345523034 +0000 UTC m=+0.433948697,LastTimestamp:2026-07-07 15:44:06.345523034 +0000 UTC m=+0.433948697,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-101.ec2.internal,}"
Jul 07 15:44:06.389422 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.389408    2566 manager.go:337] "Starting Device Plugin manager"
Jul 07 15:44:06.405748 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:06.389438    2566 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 07 15:44:06.405748 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.389448    2566 server.go:87] "Starting device plugin registration server"
Jul 07 15:44:06.405748 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.389842    2566 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 07 15:44:06.405748 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.389853    2566 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 07 15:44:06.405748 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.389931    2566 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 07 15:44:06.405748 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.390009    2566 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 07 15:44:06.405748 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.390018    2566 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 07 15:44:06.405748 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:06.390518    2566 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 07 15:44:06.405748 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:06.390559    2566 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-101.ec2.internal\" not found"
Jul 07 15:44:06.466993 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.466917    2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 07 15:44:06.468108 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.468091    2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 07 15:44:06.468165 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.468139    2566 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 07 15:44:06.468216 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.468167    2566 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 07 15:44:06.468216 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:06.468207    2566 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 07 15:44:06.470984 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.470966    2566 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 15:44:06.490091 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.490077    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 15:44:06.490940 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.490925    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-101.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 15:44:06.491032 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.490956    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-101.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 15:44:06.491032 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.490971    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-101.ec2.internal" event="NodeHasSufficientPID"
Jul 07 15:44:06.491032 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.491000    2566 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-101.ec2.internal"
Jul 07 15:44:06.500035 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.500019    2566 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-101.ec2.internal"
Jul 07 15:44:06.500112 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:06.500040    2566 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-101.ec2.internal\": node \"ip-10-0-138-101.ec2.internal\" not found"
Jul 07 15:44:06.514869 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:06.514848    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-101.ec2.internal\" not found"
Jul 07 15:44:06.568667 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.568639    2566 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-101.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-101.ec2.internal"]
Jul 07 15:44:06.568811 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.568725    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 15:44:06.569698 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.569683    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-101.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 15:44:06.569764 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.569714    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-101.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 15:44:06.569764 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.569723    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-101.ec2.internal" event="NodeHasSufficientPID"
Jul 07 15:44:06.572021 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.572009    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 15:44:06.572164 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.572150    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-101.ec2.internal"
Jul 07 15:44:06.572213 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.572178    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 15:44:06.572677 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.572661    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-101.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 15:44:06.572767 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.572686    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-101.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 15:44:06.572767 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.572692    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-101.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 15:44:06.572767 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.572700    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-101.ec2.internal" event="NodeHasSufficientPID"
Jul 07 15:44:06.572767 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.572714    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-101.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 15:44:06.572767 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.572724    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-101.ec2.internal" event="NodeHasSufficientPID"
Jul 07 15:44:06.574855 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.574839    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-101.ec2.internal"
Jul 07 15:44:06.574954 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.574868    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 15:44:06.575782 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.575768    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-101.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 15:44:06.575860 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.575790    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-101.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 15:44:06.575860 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.575815    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-101.ec2.internal" event="NodeHasSufficientPID"
Jul 07 15:44:06.601536 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:06.601508    2566 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-101.ec2.internal\" not found" node="ip-10-0-138-101.ec2.internal"
Jul 07 15:44:06.605819 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:06.605801    2566 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-101.ec2.internal\" not found" node="ip-10-0-138-101.ec2.internal"
Jul 07 15:44:06.615848 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:06.615833    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-101.ec2.internal\" not found"
Jul 07 15:44:06.629502 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.629481    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a9ccd326e5056db2ec15b8f22d5755cf-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-101.ec2.internal\" (UID: \"a9ccd326e5056db2ec15b8f22d5755cf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-101.ec2.internal"
Jul 07 15:44:06.629565 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.629506    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a9ccd326e5056db2ec15b8f22d5755cf-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-101.ec2.internal\" (UID: \"a9ccd326e5056db2ec15b8f22d5755cf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-101.ec2.internal"
Jul 07 15:44:06.629565 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.629525    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9b6c8b6d42e5b52ab4cf9f706a9be0f7-config\") pod \"kube-apiserver-proxy-ip-10-0-138-101.ec2.internal\" (UID: \"9b6c8b6d42e5b52ab4cf9f706a9be0f7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-101.ec2.internal"
Jul 07 15:44:06.716411 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:06.716385    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-101.ec2.internal\" not found"
Jul 07 15:44:06.729864 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.729812    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a9ccd326e5056db2ec15b8f22d5755cf-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-101.ec2.internal\" (UID: \"a9ccd326e5056db2ec15b8f22d5755cf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-101.ec2.internal"
Jul 07 15:44:06.729864 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.729838    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a9ccd326e5056db2ec15b8f22d5755cf-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-101.ec2.internal\" (UID: \"a9ccd326e5056db2ec15b8f22d5755cf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-101.ec2.internal"
Jul 07 15:44:06.729864 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.729856    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9b6c8b6d42e5b52ab4cf9f706a9be0f7-config\") pod \"kube-apiserver-proxy-ip-10-0-138-101.ec2.internal\" (UID: \"9b6c8b6d42e5b52ab4cf9f706a9be0f7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-101.ec2.internal"
Jul 07 15:44:06.730046 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.729911    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9b6c8b6d42e5b52ab4cf9f706a9be0f7-config\") pod \"kube-apiserver-proxy-ip-10-0-138-101.ec2.internal\" (UID: \"9b6c8b6d42e5b52ab4cf9f706a9be0f7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-101.ec2.internal"
Jul 07 15:44:06.730046 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.729960    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a9ccd326e5056db2ec15b8f22d5755cf-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-101.ec2.internal\" (UID: \"a9ccd326e5056db2ec15b8f22d5755cf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-101.ec2.internal"
Jul 07 15:44:06.730046 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.730021    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a9ccd326e5056db2ec15b8f22d5755cf-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-101.ec2.internal\" (UID: \"a9ccd326e5056db2ec15b8f22d5755cf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-101.ec2.internal"
Jul 07 15:44:06.816809 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:06.816774    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-101.ec2.internal\" not found"
Jul 07 15:44:06.909004 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.908967    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-101.ec2.internal"
Jul 07 15:44:06.909144 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:06.908968    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-101.ec2.internal"
Jul 07 15:44:06.917146 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:06.917130    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-101.ec2.internal\" not found"
Jul 07 15:44:07.019122 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:07.019019    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-101.ec2.internal\" not found"
Jul 07 15:44:07.125510 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:07.125482    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-101.ec2.internal\" not found"
Jul 07 15:44:07.233164 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:07.233130    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-101.ec2.internal\" not found"
Jul 07 15:44:07.251609 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:07.251584    2566 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 07 15:44:07.251749 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:07.251726    2566 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 07 15:44:07.333933 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:07.333858    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-101.ec2.internal\" not found"
Jul 07 15:44:07.363356 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:07.363321    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-06 15:39:06 +0000 UTC" deadline="2028-04-04 04:15:04.872267102 +0000 UTC"
Jul 07 15:44:07.363356 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:07.363350    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15276h30m57.508921267s"
Jul 07 15:44:07.385871 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:07.385850    2566 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 15:44:07.425213 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:07.425183    2566 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 15:44:07.428016 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:07.427991    2566 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-101.ec2.internal"
Jul 07 15:44:07.438126 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:07.438109    2566 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 07 15:44:07.439623 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:07.439603    2566 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-101.ec2.internal"
Jul 07 15:44:07.450456 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:07.450437    2566 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 07 15:44:07.451271 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:07.451251    2566 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 15:44:07.578483 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:07.578442    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9ccd326e5056db2ec15b8f22d5755cf.slice/crio-f35be2d12486c9e0178b8598226ddcd9c13eb90e0f464c41fe66299273f57c31 WatchSource:0}: Error finding container f35be2d12486c9e0178b8598226ddcd9c13eb90e0f464c41fe66299273f57c31: Status 404 returned error can't find the container with id f35be2d12486c9e0178b8598226ddcd9c13eb90e0f464c41fe66299273f57c31
Jul 07 15:44:07.578806 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:07.578778    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b6c8b6d42e5b52ab4cf9f706a9be0f7.slice/crio-474c0214e6d7b7697aa1dbf82fc6a00270f14dd9a550e6d695870958690c2a51 WatchSource:0}: Error finding container 474c0214e6d7b7697aa1dbf82fc6a00270f14dd9a550e6d695870958690c2a51: Status 404 returned error can't find the container with id 474c0214e6d7b7697aa1dbf82fc6a00270f14dd9a550e6d695870958690c2a51
Jul 07 15:44:07.583358 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:07.583337    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 15:44:08.071040 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.071015    2566 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 15:44:08.305998 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.305969    2566 apiserver.go:52] "Watching apiserver"
Jul 07 15:44:08.315631 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.315604    2566 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 07 15:44:08.318881 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.317970    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-zbhgc","openshift-network-diagnostics/network-check-target-tx5v9","kube-system/global-pull-secret-syncer-fvkrg","kube-system/kube-apiserver-proxy-ip-10-0-138-101.ec2.internal","openshift-cluster-node-tuning-operator/tuned-bfgft","openshift-dns/node-resolver-6cgnn","openshift-image-registry/node-ca-shpjq","openshift-network-operator/iptables-alerter-v7lzm","openshift-ovn-kubernetes/ovnkube-node-vfd6r","kube-system/konnectivity-agent-r9nv8","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lq9gt","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-101.ec2.internal","openshift-multus/multus-74zmd","openshift-multus/multus-additional-cni-plugins-zbxwc"]
Jul 07 15:44:08.321469 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.321370    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zbhgc"
Jul 07 15:44:08.321593 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:08.321512    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zbhgc" podUID="2a773f68-4a73-415d-abc9-3672aa14a254"
Jul 07 15:44:08.323765 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.323735    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tx5v9"
Jul 07 15:44:08.323887 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:08.323811    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tx5v9" podUID="e86522b9-3dc8-41c0-8d8a-40229b4795a7"
Jul 07 15:44:08.326099 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.326065    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fvkrg"
Jul 07 15:44:08.326225 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:08.326123    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fvkrg" podUID="f0d07c9b-7fac-4665-b397-d996f6781c50"
Jul 07 15:44:08.327099 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.327074    2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 07 15:44:08.328497 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.328475    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bfgft"
Jul 07 15:44:08.330712 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.330689    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 07 15:44:08.330815 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.330732    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 07 15:44:08.330885 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.330700    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-9hc66\""
Jul 07 15:44:08.331008 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.330987    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6cgnn"
Jul 07 15:44:08.333495 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.333447    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-shpjq"
Jul 07 15:44:08.333495 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.333470    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 07 15:44:08.333757 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.333735    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 07 15:44:08.334039 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.334021    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-v7mck\""
Jul 07 15:44:08.335586 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.335565    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 07 15:44:08.335857 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.335838    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 07 15:44:08.335936 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.335867    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 07 15:44:08.336029 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.335847    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-nb2d2\""
Jul 07 15:44:08.336146 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.336131    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-v7lzm"
Jul 07 15:44:08.338939 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.338497    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 07 15:44:08.338939 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.338664    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 07 15:44:08.338939 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.338707    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-f2qrz\""
Jul 07 15:44:08.338939 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.338497    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 07 15:44:08.339161 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.339009    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vfd6r"
Jul 07 15:44:08.341227 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.341206    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6e18ef02-9812-4ed6-828e-3f88d2429a7d-etc-systemd\") pod \"tuned-bfgft\" (UID: \"6e18ef02-9812-4ed6-828e-3f88d2429a7d\") " pod="openshift-cluster-node-tuning-operator/tuned-bfgft"
Jul 07 15:44:08.341328 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.341241    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6e18ef02-9812-4ed6-828e-3f88d2429a7d-sys\") pod \"tuned-bfgft\" (UID: \"6e18ef02-9812-4ed6-828e-3f88d2429a7d\") " pod="openshift-cluster-node-tuning-operator/tuned-bfgft"
Jul 07 15:44:08.341328 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.341266    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6e18ef02-9812-4ed6-828e-3f88d2429a7d-lib-modules\") pod \"tuned-bfgft\" (UID: \"6e18ef02-9812-4ed6-828e-3f88d2429a7d\") " pod="openshift-cluster-node-tuning-operator/tuned-bfgft"
Jul 07 15:44:08.341328 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.341321    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6e18ef02-9812-4ed6-828e-3f88d2429a7d-var-lib-kubelet\") pod \"tuned-bfgft\" (UID: \"6e18ef02-9812-4ed6-828e-3f88d2429a7d\") " pod="openshift-cluster-node-tuning-operator/tuned-bfgft"
Jul 07 15:44:08.341494 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.341345    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6e18ef02-9812-4ed6-828e-3f88d2429a7d-tmp\") pod \"tuned-bfgft\" (UID: \"6e18ef02-9812-4ed6-828e-3f88d2429a7d\") " pod="openshift-cluster-node-tuning-operator/tuned-bfgft"
Jul 07 15:44:08.341494 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.341371    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-724lq\" (UniqueName: \"kubernetes.io/projected/2a773f68-4a73-415d-abc9-3672aa14a254-kube-api-access-724lq\") pod \"network-metrics-daemon-zbhgc\" (UID: \"2a773f68-4a73-415d-abc9-3672aa14a254\") " pod="openshift-multus/network-metrics-daemon-zbhgc"
Jul 07 15:44:08.341494 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.341398    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f0d07c9b-7fac-4665-b397-d996f6781c50-kubelet-config\") pod \"global-pull-secret-syncer-fvkrg\" (UID: \"f0d07c9b-7fac-4665-b397-d996f6781c50\") " pod="kube-system/global-pull-secret-syncer-fvkrg"
Jul 07 15:44:08.341494 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.341457    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6e18ef02-9812-4ed6-828e-3f88d2429a7d-etc-kubernetes\") pod \"tuned-bfgft\" (UID: \"6e18ef02-9812-4ed6-828e-3f88d2429a7d\") " pod="openshift-cluster-node-tuning-operator/tuned-bfgft"
Jul 07 15:44:08.341494 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.341487    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mll9x\" (UniqueName: \"kubernetes.io/projected/6e18ef02-9812-4ed6-828e-3f88d2429a7d-kube-api-access-mll9x\") pod \"tuned-bfgft\" (UID: \"6e18ef02-9812-4ed6-828e-3f88d2429a7d\") " pod="openshift-cluster-node-tuning-operator/tuned-bfgft"
Jul 07 15:44:08.341754 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.341518    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a773f68-4a73-415d-abc9-3672aa14a254-metrics-certs\") pod \"network-metrics-daemon-zbhgc\" (UID: \"2a773f68-4a73-415d-abc9-3672aa14a254\") " pod="openshift-multus/network-metrics-daemon-zbhgc"
Jul 07 15:44:08.341754 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.341544    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f0d07c9b-7fac-4665-b397-d996f6781c50-original-pull-secret\") pod \"global-pull-secret-syncer-fvkrg\" (UID: \"f0d07c9b-7fac-4665-b397-d996f6781c50\") " pod="kube-system/global-pull-secret-syncer-fvkrg"
Jul 07 15:44:08.341754 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.341603    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6e18ef02-9812-4ed6-828e-3f88d2429a7d-etc-modprobe-d\") pod \"tuned-bfgft\" (UID: \"6e18ef02-9812-4ed6-828e-3f88d2429a7d\") " pod="openshift-cluster-node-tuning-operator/tuned-bfgft"
Jul 07 15:44:08.341754 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.341684    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6e18ef02-9812-4ed6-828e-3f88d2429a7d-run\") pod \"tuned-bfgft\" (UID: \"6e18ef02-9812-4ed6-828e-3f88d2429a7d\") " pod="openshift-cluster-node-tuning-operator/tuned-bfgft"
Jul 07 15:44:08.341754 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.341729    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6e18ef02-9812-4ed6-828e-3f88d2429a7d-etc-tuned\") pod \"tuned-bfgft\" (UID: \"6e18ef02-9812-4ed6-828e-3f88d2429a7d\") " pod="openshift-cluster-node-tuning-operator/tuned-bfgft"
Jul 07 15:44:08.342023 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.341774    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4wkf\" (UniqueName: \"kubernetes.io/projected/e86522b9-3dc8-41c0-8d8a-40229b4795a7-kube-api-access-m4wkf\") pod \"network-check-target-tx5v9\" (UID: \"e86522b9-3dc8-41c0-8d8a-40229b4795a7\") " pod="openshift-network-diagnostics/network-check-target-tx5v9"
Jul 07 15:44:08.342023 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.341800    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f0d07c9b-7fac-4665-b397-d996f6781c50-dbus\") pod \"global-pull-secret-syncer-fvkrg\" (UID: \"f0d07c9b-7fac-4665-b397-d996f6781c50\") " pod="kube-system/global-pull-secret-syncer-fvkrg"
Jul 07 15:44:08.342023 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.341809    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 07 15:44:08.342023 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.341824    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6e18ef02-9812-4ed6-828e-3f88d2429a7d-etc-sysconfig\") pod \"tuned-bfgft\" (UID: \"6e18ef02-9812-4ed6-828e-3f88d2429a7d\") " pod="openshift-cluster-node-tuning-operator/tuned-bfgft"
Jul 07 15:44:08.342023 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.341831    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 07 15:44:08.342023 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.341850    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6e18ef02-9812-4ed6-828e-3f88d2429a7d-etc-sysctl-d\") pod \"tuned-bfgft\" (UID: \"6e18ef02-9812-4ed6-828e-3f88d2429a7d\") " pod="openshift-cluster-node-tuning-operator/tuned-bfgft"
Jul 07 15:44:08.342023 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.341905    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6e18ef02-9812-4ed6-828e-3f88d2429a7d-etc-sysctl-conf\") pod \"tuned-bfgft\" (UID: \"6e18ef02-9812-4ed6-828e-3f88d2429a7d\") " pod="openshift-cluster-node-tuning-operator/tuned-bfgft"
Jul 07 15:44:08.342023 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.341946    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6e18ef02-9812-4ed6-828e-3f88d2429a7d-host\") pod \"tuned-bfgft\" (UID: \"6e18ef02-9812-4ed6-828e-3f88d2429a7d\") " pod="openshift-cluster-node-tuning-operator/tuned-bfgft"
Jul 07 15:44:08.342422 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.342106    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 07 15:44:08.342422 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.342198    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-r9nv8"
Jul 07 15:44:08.342422 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.342314    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 07 15:44:08.343084 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.342834    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 07 15:44:08.343084 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.342889    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-hswqs\""
Jul 07 15:44:08.343084 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.342916    2566 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 07 15:44:08.343084 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.343015    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 07 15:44:08.344410 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.343794    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lq9gt"
Jul 07 15:44:08.344868 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.344848    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 07 15:44:08.345109 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.345087    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-vptng\""
Jul 07 15:44:08.345109 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.345093    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 07 15:44:08.347132 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.346606    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 07 15:44:08.347132 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.346660    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-vf65g\""
Jul 07 15:44:08.347132 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.346672    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 07 15:44:08.347132 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.346608    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-74zmd"
Jul 07 15:44:08.347132 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.346930    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 07 15:44:08.349104 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.349086    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 07 15:44:08.349213 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.349192    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zbxwc"
Jul 07 15:44:08.349998 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.349583    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 07 15:44:08.349998 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.349591    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 07 15:44:08.349998 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.349718    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 07 15:44:08.349998 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.349854    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-g6dfq\""
Jul 07 15:44:08.351364 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.351338    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-ggpxr\""
Jul 07 15:44:08.351722 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.351702    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 07 15:44:08.352134 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.352099    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 07 15:44:08.369894 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.369865    2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-bw27k"
Jul 07 15:44:08.381141 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.381118    2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-bw27k"
Jul 07 15:44:08.428755 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.428698    2566 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 07 15:44:08.442329 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.442302    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b9e1b70f-f064-49bb-ab99-39d7bb57a5d1-etc-selinux\") pod \"aws-ebs-csi-driver-node-lq9gt\" (UID: \"b9e1b70f-f064-49bb-ab99-39d7bb57a5d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lq9gt"
Jul 07 15:44:08.442454 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.442340    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0e3de181-9d7a-4f84-ae08-d018e710109c-os-release\") pod \"multus-additional-cni-plugins-zbxwc\" (UID: \"0e3de181-9d7a-4f84-ae08-d018e710109c\") " pod="openshift-multus/multus-additional-cni-plugins-zbxwc"
Jul 07 15:44:08.442454 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.442376    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6e18ef02-9812-4ed6-828e-3f88d2429a7d-etc-sysctl-d\") pod \"tuned-bfgft\" (UID: \"6e18ef02-9812-4ed6-828e-3f88d2429a7d\") " pod="openshift-cluster-node-tuning-operator/tuned-bfgft"
Jul 07 15:44:08.442454 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.442401    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6e18ef02-9812-4ed6-828e-3f88d2429a7d-host\") pod \"tuned-bfgft\" (UID: \"6e18ef02-9812-4ed6-828e-3f88d2429a7d\") " pod="openshift-cluster-node-tuning-operator/tuned-bfgft"
Jul 07 15:44:08.442454 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.442425    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/22f1cba0-dd6e-4f5a-8d37-d83d1f938ace-hosts-file\") pod \"node-resolver-6cgnn\" (UID: \"22f1cba0-dd6e-4f5a-8d37-d83d1f938ace\") " pod="openshift-dns/node-resolver-6cgnn"
Jul 07 15:44:08.442681 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.442455    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb-host-cni-bin\") pod \"ovnkube-node-vfd6r\" (UID: \"b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfd6r"
Jul 07 15:44:08.442681 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.442463    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6e18ef02-9812-4ed6-828e-3f88d2429a7d-host\") pod \"tuned-bfgft\" (UID: \"6e18ef02-9812-4ed6-828e-3f88d2429a7d\") " pod="openshift-cluster-node-tuning-operator/tuned-bfgft"
Jul 07 15:44:08.442681 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.442479    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb-host-cni-netd\") pod \"ovnkube-node-vfd6r\" (UID: \"b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfd6r"
Jul 07 15:44:08.442681 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.442505    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6e18ef02-9812-4ed6-828e-3f88d2429a7d-lib-modules\") pod \"tuned-bfgft\" (UID: \"6e18ef02-9812-4ed6-828e-3f88d2429a7d\") " pod="openshift-cluster-node-tuning-operator/tuned-bfgft"
Jul 07 15:44:08.442681 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.442529    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb-ovnkube-config\") pod \"ovnkube-node-vfd6r\" (UID: \"b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfd6r"
Jul 07 15:44:08.442681 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.442540    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6e18ef02-9812-4ed6-828e-3f88d2429a7d-etc-sysctl-d\") pod \"tuned-bfgft\" (UID: \"6e18ef02-9812-4ed6-828e-3f88d2429a7d\") " pod="openshift-cluster-node-tuning-operator/tuned-bfgft"
Jul 07 15:44:08.442681 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.442549    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/69ca53b7-f45f-471b-96b0-1cff9686d44f-system-cni-dir\") pod \"multus-74zmd\" (UID: \"69ca53b7-f45f-471b-96b0-1cff9686d44f\") " pod="openshift-multus/multus-74zmd"
Jul 07 15:44:08.442681 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.442572    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/69ca53b7-f45f-471b-96b0-1cff9686d44f-os-release\") pod \"multus-74zmd\" (UID: \"69ca53b7-f45f-471b-96b0-1cff9686d44f\") " pod="openshift-multus/multus-74zmd"
Jul 07 15:44:08.442681 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.442596    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-724lq\" (UniqueName: \"kubernetes.io/projected/2a773f68-4a73-415d-abc9-3672aa14a254-kube-api-access-724lq\") pod \"network-metrics-daemon-zbhgc\" (UID: \"2a773f68-4a73-415d-abc9-3672aa14a254\") " pod="openshift-multus/network-metrics-daemon-zbhgc"
Jul 07 15:44:08.442681 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.442614    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6e18ef02-9812-4ed6-828e-3f88d2429a7d-etc-kubernetes\") pod \"tuned-bfgft\" (UID: \"6e18ef02-9812-4ed6-828e-3f88d2429a7d\") " pod="openshift-cluster-node-tuning-operator/tuned-bfgft"
Jul 07 15:44:08.442681 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.442658    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6e18ef02-9812-4ed6-828e-3f88d2429a7d-lib-modules\") pod \"tuned-bfgft\" (UID: \"6e18ef02-9812-4ed6-828e-3f88d2429a7d\") " pod="openshift-cluster-node-tuning-operator/tuned-bfgft"
Jul 07 15:44:08.443234 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.442684    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6e18ef02-9812-4ed6-828e-3f88d2429a7d-etc-kubernetes\") pod \"tuned-bfgft\" (UID: \"6e18ef02-9812-4ed6-828e-3f88d2429a7d\") " pod="openshift-cluster-node-tuning-operator/tuned-bfgft"
Jul 07 15:44:08.443234 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.442688    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb-node-log\") pod \"ovnkube-node-vfd6r\" (UID: \"b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfd6r"
Jul 07 15:44:08.443234 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.442717    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb-log-socket\") pod \"ovnkube-node-vfd6r\" (UID: \"b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfd6r"
Jul 07 15:44:08.443234 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.442737    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb-ovnkube-script-lib\") pod \"ovnkube-node-vfd6r\" (UID: \"b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfd6r"
Jul 07 15:44:08.443234 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.442772    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/69ca53b7-f45f-471b-96b0-1cff9686d44f-multus-daemon-config\") pod \"multus-74zmd\" (UID: \"69ca53b7-f45f-471b-96b0-1cff9686d44f\") " pod="openshift-multus/multus-74zmd"
Jul 07 15:44:08.443234 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.442792    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0e3de181-9d7a-4f84-ae08-d018e710109c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zbxwc\" (UID: \"0e3de181-9d7a-4f84-ae08-d018e710109c\") " pod="openshift-multus/multus-additional-cni-plugins-zbxwc"
Jul 07 15:44:08.443234 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.442814    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a773f68-4a73-415d-abc9-3672aa14a254-metrics-certs\") pod \"network-metrics-daemon-zbhgc\" (UID: \"2a773f68-4a73-415d-abc9-3672aa14a254\") " pod="openshift-multus/network-metrics-daemon-zbhgc"
Jul 07 15:44:08.443234 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.442864    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f0d07c9b-7fac-4665-b397-d996f6781c50-original-pull-secret\") pod \"global-pull-secret-syncer-fvkrg\" (UID: \"f0d07c9b-7fac-4665-b397-d996f6781c50\") " pod="kube-system/global-pull-secret-syncer-fvkrg"
Jul 07 15:44:08.443234 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:08.442956    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 15:44:08.443234 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.443006    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb-var-lib-openvswitch\") pod \"ovnkube-node-vfd6r\" (UID: \"b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfd6r"
Jul 07 15:44:08.443234 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:08.443016    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 15:44:08.443234 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:08.443027    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a773f68-4a73-415d-abc9-3672aa14a254-metrics-certs podName:2a773f68-4a73-415d-abc9-3672aa14a254 nodeName:}" failed. No retries permitted until 2026-07-07 15:44:08.943003143 +0000 UTC m=+3.031428813 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2a773f68-4a73-415d-abc9-3672aa14a254-metrics-certs") pod "network-metrics-daemon-zbhgc" (UID: "2a773f68-4a73-415d-abc9-3672aa14a254") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 15:44:08.443234 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.443056    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4e92cac1-64ea-498d-b27c-30403f6e8b81-konnectivity-ca\") pod \"konnectivity-agent-r9nv8\" (UID: \"4e92cac1-64ea-498d-b27c-30403f6e8b81\") " pod="kube-system/konnectivity-agent-r9nv8"
Jul 07 15:44:08.443234 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:08.443070    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0d07c9b-7fac-4665-b397-d996f6781c50-original-pull-secret podName:f0d07c9b-7fac-4665-b397-d996f6781c50 nodeName:}" failed. No retries permitted until 2026-07-07 15:44:08.943053261 +0000 UTC m=+3.031478917 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f0d07c9b-7fac-4665-b397-d996f6781c50-original-pull-secret") pod "global-pull-secret-syncer-fvkrg" (UID: "f0d07c9b-7fac-4665-b397-d996f6781c50") : object "kube-system"/"original-pull-secret" not registered
Jul 07 15:44:08.443234 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.443104    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b9e1b70f-f064-49bb-ab99-39d7bb57a5d1-socket-dir\") pod \"aws-ebs-csi-driver-node-lq9gt\" (UID: \"b9e1b70f-f064-49bb-ab99-39d7bb57a5d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lq9gt"
Jul 07 15:44:08.443234 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.443130    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b9e1b70f-f064-49bb-ab99-39d7bb57a5d1-registration-dir\") pod \"aws-ebs-csi-driver-node-lq9gt\" (UID: \"b9e1b70f-f064-49bb-ab99-39d7bb57a5d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lq9gt"
Jul 07 15:44:08.443959 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.443156    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/69ca53b7-f45f-471b-96b0-1cff9686d44f-cnibin\") pod \"multus-74zmd\" (UID: \"69ca53b7-f45f-471b-96b0-1cff9686d44f\") " pod="openshift-multus/multus-74zmd"
Jul 07 15:44:08.443959 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.443199    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/69ca53b7-f45f-471b-96b0-1cff9686d44f-cni-binary-copy\") pod \"multus-74zmd\" (UID: \"69ca53b7-f45f-471b-96b0-1cff9686d44f\") " pod="openshift-multus/multus-74zmd"
Jul 07 15:44:08.443959 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.443224    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/69ca53b7-f45f-471b-96b0-1cff9686d44f-host-var-lib-kubelet\") pod \"multus-74zmd\" (UID: \"69ca53b7-f45f-471b-96b0-1cff9686d44f\") " pod="openshift-multus/multus-74zmd"
Jul 07 15:44:08.443959 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.443248    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/69ca53b7-f45f-471b-96b0-1cff9686d44f-hostroot\") pod \"multus-74zmd\" (UID: \"69ca53b7-f45f-471b-96b0-1cff9686d44f\") " pod="openshift-multus/multus-74zmd"
Jul 07 15:44:08.443959 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.443273    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/69ca53b7-f45f-471b-96b0-1cff9686d44f-multus-conf-dir\") pod \"multus-74zmd\" (UID: \"69ca53b7-f45f-471b-96b0-1cff9686d44f\") " pod="openshift-multus/multus-74zmd"
Jul 07 15:44:08.443959 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.443302    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dt5k4\" (UniqueName: \"kubernetes.io/projected/0e3de181-9d7a-4f84-ae08-d018e710109c-kube-api-access-dt5k4\") pod \"multus-additional-cni-plugins-zbxwc\" (UID: \"0e3de181-9d7a-4f84-ae08-d018e710109c\") " pod="openshift-multus/multus-additional-cni-plugins-zbxwc"
Jul 07 15:44:08.443959 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.443331    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6e18ef02-9812-4ed6-828e-3f88d2429a7d-sys\") pod \"tuned-bfgft\" (UID: \"6e18ef02-9812-4ed6-828e-3f88d2429a7d\") " pod="openshift-cluster-node-tuning-operator/tuned-bfgft"
Jul 07 15:44:08.443959 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.443355    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb-run-openvswitch\") pod \"ovnkube-node-vfd6r\" (UID: \"b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfd6r"
Jul 07 15:44:08.443959 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.443383    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b9e1b70f-f064-49bb-ab99-39d7bb57a5d1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lq9gt\" (UID: \"b9e1b70f-f064-49bb-ab99-39d7bb57a5d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lq9gt"
Jul 07 15:44:08.443959 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.443399    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6e18ef02-9812-4ed6-828e-3f88d2429a7d-sys\") pod \"tuned-bfgft\" (UID: \"6e18ef02-9812-4ed6-828e-3f88d2429a7d\") " pod="openshift-cluster-node-tuning-operator/tuned-bfgft"
Jul 07 15:44:08.443959 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.443409    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79sx2\" (UniqueName: \"kubernetes.io/projected/69ca53b7-f45f-471b-96b0-1cff9686d44f-kube-api-access-79sx2\") pod \"multus-74zmd\" (UID: \"69ca53b7-f45f-471b-96b0-1cff9686d44f\") " pod="openshift-multus/multus-74zmd"
Jul 07 15:44:08.443959 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.443439    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0e3de181-9d7a-4f84-ae08-d018e710109c-system-cni-dir\") pod \"multus-additional-cni-plugins-zbxwc\" (UID: \"0e3de181-9d7a-4f84-ae08-d018e710109c\") " pod="openshift-multus/multus-additional-cni-plugins-zbxwc"
Jul 07 15:44:08.443959 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.443466    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0e3de181-9d7a-4f84-ae08-d018e710109c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zbxwc\" (UID: \"0e3de181-9d7a-4f84-ae08-d018e710109c\") " pod="openshift-multus/multus-additional-cni-plugins-zbxwc"
Jul 07 15:44:08.443959 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.443494    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f0d07c9b-7fac-4665-b397-d996f6781c50-kubelet-config\") pod \"global-pull-secret-syncer-fvkrg\" (UID: \"f0d07c9b-7fac-4665-b397-d996f6781c50\") " pod="kube-system/global-pull-secret-syncer-fvkrg"
Jul 07 15:44:08.443959 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.443511    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb-host-slash\") pod \"ovnkube-node-vfd6r\" (UID: \"b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfd6r"
Jul 07 15:44:08.443959 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.443532    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb-run-systemd\") pod \"ovnkube-node-vfd6r\" (UID: \"b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfd6r"
Jul 07 15:44:08.443959 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.443548    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/69ca53b7-f45f-471b-96b0-1cff9686d44f-host-var-lib-cni-bin\") pod \"multus-74zmd\" (UID: \"69ca53b7-f45f-471b-96b0-1cff9686d44f\") " pod="openshift-multus/multus-74zmd"
Jul 07 15:44:08.444795 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.443547    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f0d07c9b-7fac-4665-b397-d996f6781c50-kubelet-config\") pod \"global-pull-secret-syncer-fvkrg\" (UID: \"f0d07c9b-7fac-4665-b397-d996f6781c50\") " pod="kube-system/global-pull-secret-syncer-fvkrg"
Jul 07 15:44:08.444795 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.443563    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0e3de181-9d7a-4f84-ae08-d018e710109c-cni-binary-copy\") pod \"multus-additional-cni-plugins-zbxwc\" (UID: \"0e3de181-9d7a-4f84-ae08-d018e710109c\") " pod="openshift-multus/multus-additional-cni-plugins-zbxwc"
Jul 07 15:44:08.444795 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.443580    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb-host-run-ovn-kubernetes\") pod \"ovnkube-node-vfd6r\" (UID: \"b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfd6r"
Jul 07 15:44:08.444795 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.443604    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/69ca53b7-f45f-471b-96b0-1cff9686d44f-host-run-multus-certs\") pod \"multus-74zmd\" (UID: \"69ca53b7-f45f-471b-96b0-1cff9686d44f\") " pod="openshift-multus/multus-74zmd"
Jul 07 15:44:08.444795 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.443634    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4wkf\" (UniqueName: \"kubernetes.io/projected/e86522b9-3dc8-41c0-8d8a-40229b4795a7-kube-api-access-m4wkf\") pod \"network-check-target-tx5v9\" (UID: \"e86522b9-3dc8-41c0-8d8a-40229b4795a7\") " pod="openshift-network-diagnostics/network-check-target-tx5v9"
Jul 07 15:44:08.444795 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.443661    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f0d07c9b-7fac-4665-b397-d996f6781c50-dbus\") pod \"global-pull-secret-syncer-fvkrg\" (UID: \"f0d07c9b-7fac-4665-b397-d996f6781c50\") " pod="kube-system/global-pull-secret-syncer-fvkrg"
Jul 07 15:44:08.444795 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.443794    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/06e24adc-93cb-495b-94f2-d1435ae35b39-iptables-alerter-script\") pod \"iptables-alerter-v7lzm\" (UID: \"06e24adc-93cb-495b-94f2-d1435ae35b39\") " pod="openshift-network-operator/iptables-alerter-v7lzm"
Jul 07 15:44:08.444795 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.443833    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vfd6r\" (UID: \"b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfd6r"
Jul 07 15:44:08.444795 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.443872    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f0d07c9b-7fac-4665-b397-d996f6781c50-dbus\") pod \"global-pull-secret-syncer-fvkrg\" (UID: \"f0d07c9b-7fac-4665-b397-d996f6781c50\") " pod="kube-system/global-pull-secret-syncer-fvkrg"
Jul 07 15:44:08.444795 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.443866    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b9e1b70f-f064-49bb-ab99-39d7bb57a5d1-sys-fs\") pod \"aws-ebs-csi-driver-node-lq9gt\" (UID: \"b9e1b70f-f064-49bb-ab99-39d7bb57a5d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lq9gt"
Jul 07 15:44:08.444795 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.443933    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/69ca53b7-f45f-471b-96b0-1cff9686d44f-etc-kubernetes\") pod \"multus-74zmd\" (UID: \"69ca53b7-f45f-471b-96b0-1cff9686d44f\") " pod="openshift-multus/multus-74zmd"
Jul 07 15:44:08.444795 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.443952    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6e18ef02-9812-4ed6-828e-3f88d2429a7d-etc-systemd\") pod \"tuned-bfgft\" (UID: \"6e18ef02-9812-4ed6-828e-3f88d2429a7d\") " pod="openshift-cluster-node-tuning-operator/tuned-bfgft"
Jul 07 15:44:08.444795 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.443968    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6e18ef02-9812-4ed6-828e-3f88d2429a7d-var-lib-kubelet\") pod \"tuned-bfgft\" (UID: \"6e18ef02-9812-4ed6-828e-3f88d2429a7d\") " pod="openshift-cluster-node-tuning-operator/tuned-bfgft"
Jul 07 15:44:08.444795 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.443983    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqsvf\" (UniqueName: \"kubernetes.io/projected/22f1cba0-dd6e-4f5a-8d37-d83d1f938ace-kube-api-access-bqsvf\") pod \"node-resolver-6cgnn\" (UID: \"22f1cba0-dd6e-4f5a-8d37-d83d1f938ace\") " pod="openshift-dns/node-resolver-6cgnn"
Jul 07 15:44:08.444795 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.444010    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/06e24adc-93cb-495b-94f2-d1435ae35b39-host-slash\") pod \"iptables-alerter-v7lzm\" (UID: \"06e24adc-93cb-495b-94f2-d1435ae35b39\") " pod="openshift-network-operator/iptables-alerter-v7lzm"
Jul 07 15:44:08.444795 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.444044    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6e18ef02-9812-4ed6-828e-3f88d2429a7d-etc-systemd\") pod \"tuned-bfgft\" (UID: \"6e18ef02-9812-4ed6-828e-3f88d2429a7d\") " pod="openshift-cluster-node-tuning-operator/tuned-bfgft"
Jul 07 15:44:08.445542 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.444067    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf2t6\" (UniqueName: \"kubernetes.io/projected/b9e1b70f-f064-49bb-ab99-39d7bb57a5d1-kube-api-access-kf2t6\") pod \"aws-ebs-csi-driver-node-lq9gt\" (UID: \"b9e1b70f-f064-49bb-ab99-39d7bb57a5d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lq9gt"
Jul 07 15:44:08.445542 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.444066    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6e18ef02-9812-4ed6-828e-3f88d2429a7d-var-lib-kubelet\") pod \"tuned-bfgft\" (UID: \"6e18ef02-9812-4ed6-828e-3f88d2429a7d\") " pod="openshift-cluster-node-tuning-operator/tuned-bfgft"
Jul 07 15:44:08.445542 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.444088    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/69ca53b7-f45f-471b-96b0-1cff9686d44f-multus-cni-dir\") pod \"multus-74zmd\" (UID: \"69ca53b7-f45f-471b-96b0-1cff9686d44f\") " pod="openshift-multus/multus-74zmd"
Jul 07 15:44:08.445542 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.444121    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/69ca53b7-f45f-471b-96b0-1cff9686d44f-multus-socket-dir-parent\") pod \"multus-74zmd\" (UID: \"69ca53b7-f45f-471b-96b0-1cff9686d44f\") " pod="openshift-multus/multus-74zmd"
Jul 07 15:44:08.445542 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.444152    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/69ca53b7-f45f-471b-96b0-1cff9686d44f-host-var-lib-cni-multus\") pod \"multus-74zmd\" (UID: \"69ca53b7-f45f-471b-96b0-1cff9686d44f\") " pod="openshift-multus/multus-74zmd"
Jul 07 15:44:08.445542 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.444194    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb-host-kubelet\") pod \"ovnkube-node-vfd6r\" (UID: \"b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfd6r"
Jul 07 15:44:08.445542 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.444229    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb-ovn-node-metrics-cert\") pod \"ovnkube-node-vfd6r\" (UID: \"b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfd6r"
Jul 07 15:44:08.445542 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.444262    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b9e1b70f-f064-49bb-ab99-39d7bb57a5d1-device-dir\") pod \"aws-ebs-csi-driver-node-lq9gt\" (UID: \"b9e1b70f-f064-49bb-ab99-39d7bb57a5d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lq9gt"
Jul 07 15:44:08.445542 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.444297    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/69ca53b7-f45f-471b-96b0-1cff9686d44f-host-run-k8s-cni-cncf-io\") pod \"multus-74zmd\" (UID: \"69ca53b7-f45f-471b-96b0-1cff9686d44f\") " pod="openshift-multus/multus-74zmd"
Jul 07 15:44:08.445542 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.444325    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6e18ef02-9812-4ed6-828e-3f88d2429a7d-run\") pod \"tuned-bfgft\" (UID: \"6e18ef02-9812-4ed6-828e-3f88d2429a7d\") " pod="openshift-cluster-node-tuning-operator/tuned-bfgft"
Jul 07 15:44:08.445542 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.444359    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ff08ea94-65cb-4ca1-a272-3de7627d10b8-host\") pod \"node-ca-shpjq\" (UID: \"ff08ea94-65cb-4ca1-a272-3de7627d10b8\") " pod="openshift-image-registry/node-ca-shpjq"
Jul 07 15:44:08.445542 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.444420    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ff08ea94-65cb-4ca1-a272-3de7627d10b8-serviceca\") pod \"node-ca-shpjq\" (UID: \"ff08ea94-65cb-4ca1-a272-3de7627d10b8\") " pod="openshift-image-registry/node-ca-shpjq"
Jul 07 15:44:08.445542 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.444424    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6e18ef02-9812-4ed6-828e-3f88d2429a7d-run\") pod \"tuned-bfgft\" (UID: \"6e18ef02-9812-4ed6-828e-3f88d2429a7d\") " pod="openshift-cluster-node-tuning-operator/tuned-bfgft"
Jul 07 15:44:08.445542 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.444462    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6e18ef02-9812-4ed6-828e-3f88d2429a7d-etc-sysconfig\") pod \"tuned-bfgft\" (UID: \"6e18ef02-9812-4ed6-828e-3f88d2429a7d\") " pod="openshift-cluster-node-tuning-operator/tuned-bfgft"
Jul 07 15:44:08.445542 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.444494    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6e18ef02-9812-4ed6-828e-3f88d2429a7d-etc-sysctl-conf\") pod \"tuned-bfgft\" (UID: \"6e18ef02-9812-4ed6-828e-3f88d2429a7d\") " pod="openshift-cluster-node-tuning-operator/tuned-bfgft"
Jul 07 15:44:08.445542 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.444536    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb-systemd-units\") pod \"ovnkube-node-vfd6r\" (UID: \"b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfd6r"
Jul 07 15:44:08.445542 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.444562    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb-env-overrides\") pod \"ovnkube-node-vfd6r\" (UID: \"b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfd6r"
Jul 07 15:44:08.446362 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.444566    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6e18ef02-9812-4ed6-828e-3f88d2429a7d-etc-sysconfig\") pod \"tuned-bfgft\" (UID: \"6e18ef02-9812-4ed6-828e-3f88d2429a7d\") " pod="openshift-cluster-node-tuning-operator/tuned-bfgft"
Jul 07 15:44:08.446362 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.444588    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0e3de181-9d7a-4f84-ae08-d018e710109c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zbxwc\" (UID: \"0e3de181-9d7a-4f84-ae08-d018e710109c\") " pod="openshift-multus/multus-additional-cni-plugins-zbxwc"
Jul 07 15:44:08.446362 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.444616    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6e18ef02-9812-4ed6-828e-3f88d2429a7d-tmp\") pod \"tuned-bfgft\" (UID: \"6e18ef02-9812-4ed6-828e-3f88d2429a7d\") " pod="openshift-cluster-node-tuning-operator/tuned-bfgft"
Jul 07 15:44:08.446362 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.444643    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmgcl\" (UniqueName: \"kubernetes.io/projected/06e24adc-93cb-495b-94f2-d1435ae35b39-kube-api-access-mmgcl\") pod \"iptables-alerter-v7lzm\" (UID: \"06e24adc-93cb-495b-94f2-d1435ae35b39\") " pod="openshift-network-operator/iptables-alerter-v7lzm"
Jul 07 15:44:08.446362 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.444671    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb-run-ovn\") pod \"ovnkube-node-vfd6r\" (UID: \"b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfd6r"
Jul 07 15:44:08.446362 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.444678    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6e18ef02-9812-4ed6-828e-3f88d2429a7d-etc-sysctl-conf\") pod \"tuned-bfgft\" (UID: \"6e18ef02-9812-4ed6-828e-3f88d2429a7d\") " pod="openshift-cluster-node-tuning-operator/tuned-bfgft"
Jul 07 15:44:08.446362 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.444731    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mll9x\" (UniqueName: \"kubernetes.io/projected/6e18ef02-9812-4ed6-828e-3f88d2429a7d-kube-api-access-mll9x\") pod \"tuned-bfgft\" (UID: \"6e18ef02-9812-4ed6-828e-3f88d2429a7d\") " pod="openshift-cluster-node-tuning-operator/tuned-bfgft"
Jul 07 15:44:08.446362 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.444761    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22f1cba0-dd6e-4f5a-8d37-d83d1f938ace-tmp-dir\") pod \"node-resolver-6cgnn\" (UID: \"22f1cba0-dd6e-4f5a-8d37-d83d1f938ace\") " pod="openshift-dns/node-resolver-6cgnn"
Jul 07 15:44:08.446362 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.444779    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb-etc-openvswitch\") pod \"ovnkube-node-vfd6r\" (UID: \"b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfd6r"
Jul 07 15:44:08.446362 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.444795    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wvhw\" (UniqueName: \"kubernetes.io/projected/b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb-kube-api-access-7wvhw\") pod \"ovnkube-node-vfd6r\" (UID: \"b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfd6r"
Jul 07 15:44:08.446362 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.444810    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/69ca53b7-f45f-471b-96b0-1cff9686d44f-host-run-netns\") pod \"multus-74zmd\" (UID: \"69ca53b7-f45f-471b-96b0-1cff9686d44f\") " pod="openshift-multus/multus-74zmd"
Jul 07 15:44:08.446362 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.444835    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0e3de181-9d7a-4f84-ae08-d018e710109c-cnibin\") pod \"multus-additional-cni-plugins-zbxwc\" (UID: \"0e3de181-9d7a-4f84-ae08-d018e710109c\") " pod="openshift-multus/multus-additional-cni-plugins-zbxwc"
Jul 07 15:44:08.446362 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.444872    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6e18ef02-9812-4ed6-828e-3f88d2429a7d-etc-modprobe-d\") pod \"tuned-bfgft\" (UID: \"6e18ef02-9812-4ed6-828e-3f88d2429a7d\") " pod="openshift-cluster-node-tuning-operator/tuned-bfgft"
Jul 07 15:44:08.446362 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.444922    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6e18ef02-9812-4ed6-828e-3f88d2429a7d-etc-tuned\") pod \"tuned-bfgft\" (UID: \"6e18ef02-9812-4ed6-828e-3f88d2429a7d\") " pod="openshift-cluster-node-tuning-operator/tuned-bfgft"
Jul 07 15:44:08.446362 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.444953    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fczmt\" (UniqueName: \"kubernetes.io/projected/ff08ea94-65cb-4ca1-a272-3de7627d10b8-kube-api-access-fczmt\") pod \"node-ca-shpjq\" (UID: \"ff08ea94-65cb-4ca1-a272-3de7627d10b8\") " pod="openshift-image-registry/node-ca-shpjq"
Jul 07 15:44:08.446362 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.444981    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb-host-run-netns\") pod \"ovnkube-node-vfd6r\" (UID: \"b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfd6r"
Jul 07 15:44:08.446362 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.444988    2566 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 07 15:44:08.447017 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.445007    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4e92cac1-64ea-498d-b27c-30403f6e8b81-agent-certs\") pod \"konnectivity-agent-r9nv8\" (UID: \"4e92cac1-64ea-498d-b27c-30403f6e8b81\") " pod="kube-system/konnectivity-agent-r9nv8"
Jul 07 15:44:08.447017 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.445018    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6e18ef02-9812-4ed6-828e-3f88d2429a7d-etc-modprobe-d\") pod \"tuned-bfgft\" (UID: \"6e18ef02-9812-4ed6-828e-3f88d2429a7d\") " pod="openshift-cluster-node-tuning-operator/tuned-bfgft"
Jul 07 15:44:08.449104 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.449079    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6e18ef02-9812-4ed6-828e-3f88d2429a7d-etc-tuned\") pod \"tuned-bfgft\" (UID: \"6e18ef02-9812-4ed6-828e-3f88d2429a7d\") " pod="openshift-cluster-node-tuning-operator/tuned-bfgft"
Jul 07 15:44:08.449104 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.449096    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6e18ef02-9812-4ed6-828e-3f88d2429a7d-tmp\") pod \"tuned-bfgft\" (UID: \"6e18ef02-9812-4ed6-828e-3f88d2429a7d\") " pod="openshift-cluster-node-tuning-operator/tuned-bfgft"
Jul 07 15:44:08.455017 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:08.454999    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 15:44:08.455017 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:08.455021    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 15:44:08.455167 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:08.455032    2566 projected.go:196] Error preparing data for projected volume kube-api-access-m4wkf for pod openshift-network-diagnostics/network-check-target-tx5v9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 15:44:08.455167 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:08.455096    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e86522b9-3dc8-41c0-8d8a-40229b4795a7-kube-api-access-m4wkf podName:e86522b9-3dc8-41c0-8d8a-40229b4795a7 nodeName:}" failed. No retries permitted until 2026-07-07 15:44:08.955076328 +0000 UTC m=+3.043501982 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-m4wkf" (UniqueName: "kubernetes.io/projected/e86522b9-3dc8-41c0-8d8a-40229b4795a7-kube-api-access-m4wkf") pod "network-check-target-tx5v9" (UID: "e86522b9-3dc8-41c0-8d8a-40229b4795a7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 15:44:08.458231 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.458208    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-724lq\" (UniqueName: \"kubernetes.io/projected/2a773f68-4a73-415d-abc9-3672aa14a254-kube-api-access-724lq\") pod \"network-metrics-daemon-zbhgc\" (UID: \"2a773f68-4a73-415d-abc9-3672aa14a254\") " pod="openshift-multus/network-metrics-daemon-zbhgc"
Jul 07 15:44:08.459319 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.459298    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mll9x\" (UniqueName: \"kubernetes.io/projected/6e18ef02-9812-4ed6-828e-3f88d2429a7d-kube-api-access-mll9x\") pod \"tuned-bfgft\" (UID: \"6e18ef02-9812-4ed6-828e-3f88d2429a7d\") " pod="openshift-cluster-node-tuning-operator/tuned-bfgft"
Jul 07 15:44:08.473833 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.473781    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-101.ec2.internal" event={"ID":"a9ccd326e5056db2ec15b8f22d5755cf","Type":"ContainerStarted","Data":"f35be2d12486c9e0178b8598226ddcd9c13eb90e0f464c41fe66299273f57c31"}
Jul 07 15:44:08.475082 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.475024    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-101.ec2.internal" event={"ID":"9b6c8b6d42e5b52ab4cf9f706a9be0f7","Type":"ContainerStarted","Data":"474c0214e6d7b7697aa1dbf82fc6a00270f14dd9a550e6d695870958690c2a51"}
Jul 07 15:44:08.545495 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.545461    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/06e24adc-93cb-495b-94f2-d1435ae35b39-iptables-alerter-script\") pod \"iptables-alerter-v7lzm\" (UID: \"06e24adc-93cb-495b-94f2-d1435ae35b39\") " pod="openshift-network-operator/iptables-alerter-v7lzm"
Jul 07 15:44:08.545670 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.545510    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vfd6r\" (UID: \"b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfd6r"
Jul 07 15:44:08.545670 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.545541    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b9e1b70f-f064-49bb-ab99-39d7bb57a5d1-sys-fs\") pod \"aws-ebs-csi-driver-node-lq9gt\" (UID: \"b9e1b70f-f064-49bb-ab99-39d7bb57a5d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lq9gt"
Jul 07 15:44:08.545670 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.545567    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/69ca53b7-f45f-471b-96b0-1cff9686d44f-etc-kubernetes\") pod \"multus-74zmd\" (UID: \"69ca53b7-f45f-471b-96b0-1cff9686d44f\") " pod="openshift-multus/multus-74zmd"
Jul 07 15:44:08.545670 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.545594    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqsvf\" (UniqueName: \"kubernetes.io/projected/22f1cba0-dd6e-4f5a-8d37-d83d1f938ace-kube-api-access-bqsvf\") pod \"node-resolver-6cgnn\" (UID: \"22f1cba0-dd6e-4f5a-8d37-d83d1f938ace\") " pod="openshift-dns/node-resolver-6cgnn"
Jul 07 15:44:08.545670 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.545620    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/06e24adc-93cb-495b-94f2-d1435ae35b39-host-slash\") pod \"iptables-alerter-v7lzm\" (UID: \"06e24adc-93cb-495b-94f2-d1435ae35b39\") " pod="openshift-network-operator/iptables-alerter-v7lzm"
Jul 07 15:44:08.545670 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.545642    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf2t6\" (UniqueName: \"kubernetes.io/projected/b9e1b70f-f064-49bb-ab99-39d7bb57a5d1-kube-api-access-kf2t6\") pod \"aws-ebs-csi-driver-node-lq9gt\" (UID: \"b9e1b70f-f064-49bb-ab99-39d7bb57a5d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lq9gt"
Jul 07 15:44:08.545670 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.545661    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/69ca53b7-f45f-471b-96b0-1cff9686d44f-multus-cni-dir\") pod \"multus-74zmd\" (UID: \"69ca53b7-f45f-471b-96b0-1cff9686d44f\") " pod="openshift-multus/multus-74zmd"
Jul 07 15:44:08.546138 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.545676    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/69ca53b7-f45f-471b-96b0-1cff9686d44f-multus-socket-dir-parent\") pod \"multus-74zmd\" (UID: \"69ca53b7-f45f-471b-96b0-1cff9686d44f\") " pod="openshift-multus/multus-74zmd"
Jul 07 15:44:08.546138 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.545690    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/69ca53b7-f45f-471b-96b0-1cff9686d44f-host-var-lib-cni-multus\") pod \"multus-74zmd\" (UID: \"69ca53b7-f45f-471b-96b0-1cff9686d44f\") " pod="openshift-multus/multus-74zmd"
Jul 07 15:44:08.546138 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.545705    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb-host-kubelet\") pod \"ovnkube-node-vfd6r\" (UID: \"b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfd6r"
Jul 07 15:44:08.546138 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.545727    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb-ovn-node-metrics-cert\") pod \"ovnkube-node-vfd6r\" (UID: \"b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfd6r"
Jul 07 15:44:08.546138 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.545756    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b9e1b70f-f064-49bb-ab99-39d7bb57a5d1-device-dir\") pod \"aws-ebs-csi-driver-node-lq9gt\" (UID: \"b9e1b70f-f064-49bb-ab99-39d7bb57a5d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lq9gt"
Jul 07 15:44:08.546138 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.545772    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/69ca53b7-f45f-471b-96b0-1cff9686d44f-host-run-k8s-cni-cncf-io\") pod \"multus-74zmd\" (UID: \"69ca53b7-f45f-471b-96b0-1cff9686d44f\") " pod="openshift-multus/multus-74zmd"
Jul 07 15:44:08.546138 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.545790    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ff08ea94-65cb-4ca1-a272-3de7627d10b8-host\") pod \"node-ca-shpjq\" (UID: \"ff08ea94-65cb-4ca1-a272-3de7627d10b8\") " pod="openshift-image-registry/node-ca-shpjq"
Jul 07 15:44:08.546138 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.545809    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ff08ea94-65cb-4ca1-a272-3de7627d10b8-serviceca\") pod \"node-ca-shpjq\" (UID: \"ff08ea94-65cb-4ca1-a272-3de7627d10b8\") " pod="openshift-image-registry/node-ca-shpjq"
Jul 07 15:44:08.546138 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.545837    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb-systemd-units\") pod \"ovnkube-node-vfd6r\" (UID: \"b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfd6r"
Jul 07 15:44:08.546138 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.545856    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb-env-overrides\") pod \"ovnkube-node-vfd6r\" (UID: \"b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfd6r"
Jul 07 15:44:08.546138 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.545872    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0e3de181-9d7a-4f84-ae08-d018e710109c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zbxwc\" (UID: \"0e3de181-9d7a-4f84-ae08-d018e710109c\") " pod="openshift-multus/multus-additional-cni-plugins-zbxwc"
Jul 07 15:44:08.546138 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.545895    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmgcl\" (UniqueName: \"kubernetes.io/projected/06e24adc-93cb-495b-94f2-d1435ae35b39-kube-api-access-mmgcl\") pod \"iptables-alerter-v7lzm\" (UID: \"06e24adc-93cb-495b-94f2-d1435ae35b39\") " pod="openshift-network-operator/iptables-alerter-v7lzm"
Jul 07 15:44:08.546138 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.545920    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb-run-ovn\") pod \"ovnkube-node-vfd6r\" (UID: \"b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfd6r"
Jul 07 15:44:08.546138 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.545955    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22f1cba0-dd6e-4f5a-8d37-d83d1f938ace-tmp-dir\") pod \"node-resolver-6cgnn\" (UID: \"22f1cba0-dd6e-4f5a-8d37-d83d1f938ace\") " pod="openshift-dns/node-resolver-6cgnn"
Jul 07 15:44:08.546138 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.545972    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb-etc-openvswitch\") pod \"ovnkube-node-vfd6r\" (UID: \"b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfd6r"
Jul 07 15:44:08.546138 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.545998    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wvhw\" (UniqueName: \"kubernetes.io/projected/b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb-kube-api-access-7wvhw\") pod \"ovnkube-node-vfd6r\" (UID: \"b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfd6r"
Jul 07 15:44:08.546138 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.546016    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/69ca53b7-f45f-471b-96b0-1cff9686d44f-host-run-netns\") pod \"multus-74zmd\" (UID: \"69ca53b7-f45f-471b-96b0-1cff9686d44f\") " pod="openshift-multus/multus-74zmd"
Jul 07 15:44:08.546979 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.546048    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0e3de181-9d7a-4f84-ae08-d018e710109c-cnibin\") pod \"multus-additional-cni-plugins-zbxwc\" (UID: \"0e3de181-9d7a-4f84-ae08-d018e710109c\") " pod="openshift-multus/multus-additional-cni-plugins-zbxwc"
Jul 07 15:44:08.546979 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.546069    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fczmt\" (UniqueName: \"kubernetes.io/projected/ff08ea94-65cb-4ca1-a272-3de7627d10b8-kube-api-access-fczmt\") pod \"node-ca-shpjq\" (UID: \"ff08ea94-65cb-4ca1-a272-3de7627d10b8\") " pod="openshift-image-registry/node-ca-shpjq"
Jul 07 15:44:08.546979 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.546094    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb-host-run-netns\") pod \"ovnkube-node-vfd6r\" (UID: \"b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfd6r"
Jul 07 15:44:08.546979 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.546144    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4e92cac1-64ea-498d-b27c-30403f6e8b81-agent-certs\") pod \"konnectivity-agent-r9nv8\" (UID: \"4e92cac1-64ea-498d-b27c-30403f6e8b81\") " pod="kube-system/konnectivity-agent-r9nv8"
Jul 07 15:44:08.546979 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.546170    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b9e1b70f-f064-49bb-ab99-39d7bb57a5d1-etc-selinux\") pod \"aws-ebs-csi-driver-node-lq9gt\" (UID: \"b9e1b70f-f064-49bb-ab99-39d7bb57a5d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lq9gt"
Jul 07 15:44:08.546979 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.546183    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/06e24adc-93cb-495b-94f2-d1435ae35b39-iptables-alerter-script\") pod \"iptables-alerter-v7lzm\" (UID: \"06e24adc-93cb-495b-94f2-d1435ae35b39\") " pod="openshift-network-operator/iptables-alerter-v7lzm"
Jul 07 15:44:08.546979 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.546197    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0e3de181-9d7a-4f84-ae08-d018e710109c-os-release\") pod \"multus-additional-cni-plugins-zbxwc\" (UID: \"0e3de181-9d7a-4f84-ae08-d018e710109c\") " pod="openshift-multus/multus-additional-cni-plugins-zbxwc"
Jul 07 15:44:08.546979 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.546229    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/22f1cba0-dd6e-4f5a-8d37-d83d1f938ace-hosts-file\") pod \"node-resolver-6cgnn\" (UID: \"22f1cba0-dd6e-4f5a-8d37-d83d1f938ace\") " pod="openshift-dns/node-resolver-6cgnn"
Jul 07 15:44:08.546979 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.546260    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb-host-cni-bin\") pod \"ovnkube-node-vfd6r\" (UID: \"b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfd6r"
Jul 07 15:44:08.546979 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.546286    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb-host-cni-netd\") pod \"ovnkube-node-vfd6r\" (UID: \"b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfd6r"
Jul 07 15:44:08.546979 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.546290    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vfd6r\" (UID: \"b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfd6r"
Jul 07 15:44:08.546979 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.546290    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb-etc-openvswitch\") pod \"ovnkube-node-vfd6r\" (UID: \"b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfd6r"
Jul 07 15:44:08.546979 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.546186    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/06e24adc-93cb-495b-94f2-d1435ae35b39-host-slash\") pod \"iptables-alerter-v7lzm\" (UID: \"06e24adc-93cb-495b-94f2-d1435ae35b39\") " pod="openshift-network-operator/iptables-alerter-v7lzm"
Jul 07 15:44:08.546979 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.546312    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb-ovnkube-config\") pod \"ovnkube-node-vfd6r\" (UID: \"b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfd6r"
Jul 07 15:44:08.546979 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.546339    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/69ca53b7-f45f-471b-96b0-1cff9686d44f-system-cni-dir\") pod \"multus-74zmd\" (UID: \"69ca53b7-f45f-471b-96b0-1cff9686d44f\") " pod="openshift-multus/multus-74zmd"
Jul 07 15:44:08.546979 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.546351    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ff08ea94-65cb-4ca1-a272-3de7627d10b8-host\") pod \"node-ca-shpjq\" (UID: \"ff08ea94-65cb-4ca1-a272-3de7627d10b8\") " pod="openshift-image-registry/node-ca-shpjq"
Jul 07 15:44:08.546979 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.546365    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/69ca53b7-f45f-471b-96b0-1cff9686d44f-os-release\") pod \"multus-74zmd\" (UID: \"69ca53b7-f45f-471b-96b0-1cff9686d44f\") " pod="openshift-multus/multus-74zmd"
Jul 07 15:44:08.547715 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.546392    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb-node-log\") pod \"ovnkube-node-vfd6r\" (UID: \"b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfd6r"
Jul 07 15:44:08.547715 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.546413    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b9e1b70f-f064-49bb-ab99-39d7bb57a5d1-sys-fs\") pod \"aws-ebs-csi-driver-node-lq9gt\" (UID: \"b9e1b70f-f064-49bb-ab99-39d7bb57a5d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lq9gt"
Jul 07 15:44:08.547715 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.546417    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb-log-socket\") pod \"ovnkube-node-vfd6r\" (UID: \"b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfd6r"
Jul 07 15:44:08.547715 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.546464    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb-log-socket\") pod \"ovnkube-node-vfd6r\" (UID: \"b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfd6r"
Jul 07 15:44:08.547715 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.546476    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/69ca53b7-f45f-471b-96b0-1cff9686d44f-multus-cni-dir\") pod \"multus-74zmd\" (UID: \"69ca53b7-f45f-471b-96b0-1cff9686d44f\") " pod="openshift-multus/multus-74zmd"
Jul 07 15:44:08.547715 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.546456    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb-ovnkube-script-lib\") pod \"ovnkube-node-vfd6r\" (UID: \"b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfd6r"
Jul 07 15:44:08.547715 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.546522    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/69ca53b7-f45f-471b-96b0-1cff9686d44f-multus-daemon-config\") pod \"multus-74zmd\" (UID: \"69ca53b7-f45f-471b-96b0-1cff9686d44f\") " pod="openshift-multus/multus-74zmd"
Jul 07 15:44:08.547715 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.546551    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0e3de181-9d7a-4f84-ae08-d018e710109c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zbxwc\" (UID: \"0e3de181-9d7a-4f84-ae08-d018e710109c\") " pod="openshift-multus/multus-additional-cni-plugins-zbxwc"
Jul 07 15:44:08.547715 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.546608    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb-var-lib-openvswitch\") pod \"ovnkube-node-vfd6r\" (UID: \"b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfd6r"
Jul 07 15:44:08.547715 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.546637    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4e92cac1-64ea-498d-b27c-30403f6e8b81-konnectivity-ca\") pod \"konnectivity-agent-r9nv8\" (UID: \"4e92cac1-64ea-498d-b27c-30403f6e8b81\") " pod="kube-system/konnectivity-agent-r9nv8"
Jul 07 15:44:08.547715 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.546664    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b9e1b70f-f064-49bb-ab99-39d7bb57a5d1-socket-dir\") pod \"aws-ebs-csi-driver-node-lq9gt\" (UID: \"b9e1b70f-f064-49bb-ab99-39d7bb57a5d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lq9gt"
Jul 07 15:44:08.547715 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.546691    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b9e1b70f-f064-49bb-ab99-39d7bb57a5d1-registration-dir\") pod \"aws-ebs-csi-driver-node-lq9gt\" (UID: \"b9e1b70f-f064-49bb-ab99-39d7bb57a5d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lq9gt"
Jul 07 15:44:08.547715 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.546703    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/69ca53b7-f45f-471b-96b0-1cff9686d44f-host-run-netns\") pod \"multus-74zmd\" (UID: \"69ca53b7-f45f-471b-96b0-1cff9686d44f\") " pod="openshift-multus/multus-74zmd"
Jul 07 15:44:08.547715 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.546716    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/69ca53b7-f45f-471b-96b0-1cff9686d44f-cnibin\") pod \"multus-74zmd\" (UID: \"69ca53b7-f45f-471b-96b0-1cff9686d44f\") " pod="openshift-multus/multus-74zmd"
Jul 07 15:44:08.547715 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.546744    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/69ca53b7-f45f-471b-96b0-1cff9686d44f-cni-binary-copy\") pod \"multus-74zmd\" (UID: \"69ca53b7-f45f-471b-96b0-1cff9686d44f\") " pod="openshift-multus/multus-74zmd"
Jul 07 15:44:08.547715 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.546753    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0e3de181-9d7a-4f84-ae08-d018e710109c-cnibin\") pod \"multus-additional-cni-plugins-zbxwc\" (UID: \"0e3de181-9d7a-4f84-ae08-d018e710109c\") " pod="openshift-multus/multus-additional-cni-plugins-zbxwc"
Jul 07 15:44:08.547715 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.546771    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/69ca53b7-f45f-471b-96b0-1cff9686d44f-host-var-lib-kubelet\") pod \"multus-74zmd\" (UID: \"69ca53b7-f45f-471b-96b0-1cff9686d44f\") " pod="openshift-multus/multus-74zmd"
Jul 07 15:44:08.547715 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.546796    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/69ca53b7-f45f-471b-96b0-1cff9686d44f-hostroot\") pod \"multus-74zmd\" (UID: \"69ca53b7-f45f-471b-96b0-1cff9686d44f\") " pod="openshift-multus/multus-74zmd"
Jul 07 15:44:08.548561 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.546821    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/69ca53b7-f45f-471b-96b0-1cff9686d44f-multus-conf-dir\") pod \"multus-74zmd\" (UID: \"69ca53b7-f45f-471b-96b0-1cff9686d44f\") " pod="openshift-multus/multus-74zmd"
Jul 07 15:44:08.548561 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.546848    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt5k4\" (UniqueName: \"kubernetes.io/projected/0e3de181-9d7a-4f84-ae08-d018e710109c-kube-api-access-dt5k4\") pod \"multus-additional-cni-plugins-zbxwc\" (UID: \"0e3de181-9d7a-4f84-ae08-d018e710109c\") " pod="openshift-multus/multus-additional-cni-plugins-zbxwc"
Jul 07 15:44:08.548561 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.546851    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0e3de181-9d7a-4f84-ae08-d018e710109c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zbxwc\" (UID: \"0e3de181-9d7a-4f84-ae08-d018e710109c\") " pod="openshift-multus/multus-additional-cni-plugins-zbxwc"
Jul 07 15:44:08.548561 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.546877    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb-run-openvswitch\") pod \"ovnkube-node-vfd6r\" (UID: \"b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfd6r"
Jul 07 15:44:08.548561 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.546905    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b9e1b70f-f064-49bb-ab99-39d7bb57a5d1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lq9gt\" (UID: \"b9e1b70f-f064-49bb-ab99-39d7bb57a5d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lq9gt"
Jul 07 15:44:08.548561 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.546934    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79sx2\" (UniqueName: \"kubernetes.io/projected/69ca53b7-f45f-471b-96b0-1cff9686d44f-kube-api-access-79sx2\") pod \"multus-74zmd\" (UID: \"69ca53b7-f45f-471b-96b0-1cff9686d44f\") " pod="openshift-multus/multus-74zmd"
Jul 07 15:44:08.548561 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.546958    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb-host-run-netns\") pod \"ovnkube-node-vfd6r\" (UID: \"b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfd6r"
Jul 07 15:44:08.548561 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.546983    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0e3de181-9d7a-4f84-ae08-d018e710109c-system-cni-dir\") pod \"multus-additional-cni-plugins-zbxwc\" (UID: \"0e3de181-9d7a-4f84-ae08-d018e710109c\") " pod="openshift-multus/multus-additional-cni-plugins-zbxwc"
Jul 07 15:44:08.548561 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.547011    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0e3de181-9d7a-4f84-ae08-d018e710109c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zbxwc\" (UID: \"0e3de181-9d7a-4f84-ae08-d018e710109c\") " pod="openshift-multus/multus-additional-cni-plugins-zbxwc"
Jul 07 15:44:08.548561 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.547019    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/69ca53b7-f45f-471b-96b0-1cff9686d44f-host-run-k8s-cni-cncf-io\") pod \"multus-74zmd\" (UID: \"69ca53b7-f45f-471b-96b0-1cff9686d44f\") " pod="openshift-multus/multus-74zmd"
Jul 07 15:44:08.548561 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.547047    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb-host-slash\") pod \"ovnkube-node-vfd6r\" (UID: \"b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfd6r"
Jul 07 15:44:08.548561 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.547049    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb-ovnkube-script-lib\") pod \"ovnkube-node-vfd6r\" (UID: \"b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfd6r"
Jul 07 15:44:08.548561 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.547658    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/69ca53b7-f45f-471b-96b0-1cff9686d44f-multus-socket-dir-parent\") pod \"multus-74zmd\" (UID: \"69ca53b7-f45f-471b-96b0-1cff9686d44f\") " pod="openshift-multus/multus-74zmd"
Jul 07 15:44:08.548561 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.547725    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb-run-ovn\") pod \"ovnkube-node-vfd6r\" (UID: \"b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfd6r"
Jul 07 15:44:08.548561 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.547802    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/69ca53b7-f45f-471b-96b0-1cff9686d44f-host-var-lib-cni-multus\") pod \"multus-74zmd\" (UID: \"69ca53b7-f45f-471b-96b0-1cff9686d44f\") " pod="openshift-multus/multus-74zmd"
Jul 07 15:44:08.548561 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.547857    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb-run-systemd\") pod \"ovnkube-node-vfd6r\" (UID: \"b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfd6r"
Jul 07 15:44:08.548561 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.547946    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/69ca53b7-f45f-471b-96b0-1cff9686d44f-host-var-lib-cni-bin\") pod \"multus-74zmd\" (UID: \"69ca53b7-f45f-471b-96b0-1cff9686d44f\") " pod="openshift-multus/multus-74zmd"
Jul 07 15:44:08.549421 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.548025    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/69ca53b7-f45f-471b-96b0-1cff9686d44f-host-var-lib-cni-bin\") pod \"multus-74zmd\" (UID: \"69ca53b7-f45f-471b-96b0-1cff9686d44f\") " pod="openshift-multus/multus-74zmd"
Jul 07 15:44:08.549421 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.548105    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb-host-kubelet\") pod \"ovnkube-node-vfd6r\" (UID: \"b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfd6r"
Jul 07 15:44:08.549421 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.548116    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22f1cba0-dd6e-4f5a-8d37-d83d1f938ace-tmp-dir\") pod \"node-resolver-6cgnn\" (UID: \"22f1cba0-dd6e-4f5a-8d37-d83d1f938ace\") " pod="openshift-dns/node-resolver-6cgnn"
Jul 07 15:44:08.549421 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.548172    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb-run-systemd\") pod \"ovnkube-node-vfd6r\" (UID: \"b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfd6r"
Jul 07 15:44:08.549421 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.548262    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0e3de181-9d7a-4f84-ae08-d018e710109c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zbxwc\" (UID: \"0e3de181-9d7a-4f84-ae08-d018e710109c\") " pod="openshift-multus/multus-additional-cni-plugins-zbxwc"
Jul 07 15:44:08.549421 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.548350    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ff08ea94-65cb-4ca1-a272-3de7627d10b8-serviceca\") pod \"node-ca-shpjq\" (UID: \"ff08ea94-65cb-4ca1-a272-3de7627d10b8\") " pod="openshift-image-registry/node-ca-shpjq"
Jul 07 15:44:08.549421 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.548370    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb-host-cni-bin\") pod \"ovnkube-node-vfd6r\" (UID: \"b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfd6r"
Jul 07 15:44:08.549421 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.548412    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/69ca53b7-f45f-471b-96b0-1cff9686d44f-etc-kubernetes\") pod \"multus-74zmd\" (UID: \"69ca53b7-f45f-471b-96b0-1cff9686d44f\") " pod="openshift-multus/multus-74zmd"
Jul 07 15:44:08.549421 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.548492    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b9e1b70f-f064-49bb-ab99-39d7bb57a5d1-etc-selinux\") pod \"aws-ebs-csi-driver-node-lq9gt\" (UID: \"b9e1b70f-f064-49bb-ab99-39d7bb57a5d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lq9gt"
Jul 07 15:44:08.549421 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.548563    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/22f1cba0-dd6e-4f5a-8d37-d83d1f938ace-hosts-file\") pod \"node-resolver-6cgnn\" (UID: \"22f1cba0-dd6e-4f5a-8d37-d83d1f938ace\") " pod="openshift-dns/node-resolver-6cgnn"
Jul 07 15:44:08.549421 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.548740    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b9e1b70f-f064-49bb-ab99-39d7bb57a5d1-device-dir\") pod \"aws-ebs-csi-driver-node-lq9gt\" (UID: \"b9e1b70f-f064-49bb-ab99-39d7bb57a5d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lq9gt"
Jul 07 15:44:08.549421 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.548908    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b9e1b70f-f064-49bb-ab99-39d7bb57a5d1-socket-dir\") pod \"aws-ebs-csi-driver-node-lq9gt\" (UID: \"b9e1b70f-f064-49bb-ab99-39d7bb57a5d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lq9gt"
Jul 07 15:44:08.549421 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.549093    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0e3de181-9d7a-4f84-ae08-d018e710109c-os-release\") pod \"multus-additional-cni-plugins-zbxwc\" (UID: \"0e3de181-9d7a-4f84-ae08-d018e710109c\") " pod="openshift-multus/multus-additional-cni-plugins-zbxwc"
Jul 07 15:44:08.549421 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.549392    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb-var-lib-openvswitch\") pod \"ovnkube-node-vfd6r\" (UID: \"b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfd6r"
Jul 07 15:44:08.550108 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.549481    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b9e1b70f-f064-49bb-ab99-39d7bb57a5d1-registration-dir\") pod \"aws-ebs-csi-driver-node-lq9gt\" (UID: \"b9e1b70f-f064-49bb-ab99-39d7bb57a5d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lq9gt"
Jul 07 15:44:08.550108 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.549523    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/69ca53b7-f45f-471b-96b0-1cff9686d44f-hostroot\") pod \"multus-74zmd\" (UID: \"69ca53b7-f45f-471b-96b0-1cff9686d44f\") " pod="openshift-multus/multus-74zmd"
Jul 07 15:44:08.550108 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.549612    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/69ca53b7-f45f-471b-96b0-1cff9686d44f-multus-conf-dir\") pod \"multus-74zmd\" (UID: \"69ca53b7-f45f-471b-96b0-1cff9686d44f\") " pod="openshift-multus/multus-74zmd"
Jul 07 15:44:08.550108 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.549729    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/69ca53b7-f45f-471b-96b0-1cff9686d44f-cni-binary-copy\") pod \"multus-74zmd\" (UID: \"69ca53b7-f45f-471b-96b0-1cff9686d44f\") " pod="openshift-multus/multus-74zmd"
Jul 07 15:44:08.550108 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.549955    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb-systemd-units\") pod \"ovnkube-node-vfd6r\" (UID: \"b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfd6r"
Jul 07 15:44:08.550108 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.550047    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb-run-openvswitch\") pod \"ovnkube-node-vfd6r\" (UID: \"b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfd6r"
Jul 07 15:44:08.550381 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.550143    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/69ca53b7-f45f-471b-96b0-1cff9686d44f-multus-daemon-config\") pod \"multus-74zmd\" (UID: \"69ca53b7-f45f-471b-96b0-1cff9686d44f\") " pod="openshift-multus/multus-74zmd"
Jul 07 15:44:08.550381 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.550243    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b9e1b70f-f064-49bb-ab99-39d7bb57a5d1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lq9gt\" (UID: \"b9e1b70f-f064-49bb-ab99-39d7bb57a5d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lq9gt"
Jul 07 15:44:08.550381 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.550243    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/69ca53b7-f45f-471b-96b0-1cff9686d44f-os-release\") pod \"multus-74zmd\" (UID: \"69ca53b7-f45f-471b-96b0-1cff9686d44f\") " pod="openshift-multus/multus-74zmd"
Jul 07 15:44:08.550381 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.550259    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0e3de181-9d7a-4f84-ae08-d018e710109c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zbxwc\" (UID: \"0e3de181-9d7a-4f84-ae08-d018e710109c\") " pod="openshift-multus/multus-additional-cni-plugins-zbxwc"
Jul 07 15:44:08.550381 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.550287    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb-env-overrides\") pod \"ovnkube-node-vfd6r\" (UID: \"b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfd6r"
Jul 07 15:44:08.550381 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.550292    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb-host-cni-netd\") pod \"ovnkube-node-vfd6r\" (UID: \"b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfd6r"
Jul 07 15:44:08.550381 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.550308    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0e3de181-9d7a-4f84-ae08-d018e710109c-cni-binary-copy\") pod \"multus-additional-cni-plugins-zbxwc\" (UID: \"0e3de181-9d7a-4f84-ae08-d018e710109c\") " pod="openshift-multus/multus-additional-cni-plugins-zbxwc"
Jul 07 15:44:08.550381 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.550346    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb-host-run-ovn-kubernetes\") pod \"ovnkube-node-vfd6r\" (UID: \"b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfd6r"
Jul 07 15:44:08.550381 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.550382    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/69ca53b7-f45f-471b-96b0-1cff9686d44f-host-run-multus-certs\") pod \"multus-74zmd\" (UID: \"69ca53b7-f45f-471b-96b0-1cff9686d44f\") " pod="openshift-multus/multus-74zmd"
Jul 07 15:44:08.550836 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.550382    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb-host-slash\") pod \"ovnkube-node-vfd6r\" (UID: \"b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfd6r"
Jul 07 15:44:08.550836 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.550550    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/69ca53b7-f45f-471b-96b0-1cff9686d44f-system-cni-dir\") pod \"multus-74zmd\" (UID: \"69ca53b7-f45f-471b-96b0-1cff9686d44f\") " pod="openshift-multus/multus-74zmd"
Jul 07 15:44:08.550836 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.550611    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0e3de181-9d7a-4f84-ae08-d018e710109c-system-cni-dir\") pod \"multus-additional-cni-plugins-zbxwc\" (UID: \"0e3de181-9d7a-4f84-ae08-d018e710109c\") " pod="openshift-multus/multus-additional-cni-plugins-zbxwc"
Jul 07 15:44:08.550836 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.550653    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb-node-log\") pod \"ovnkube-node-vfd6r\" (UID: \"b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfd6r"
Jul 07 15:44:08.550836 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.550721    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb-host-run-ovn-kubernetes\") pod \"ovnkube-node-vfd6r\" (UID: \"b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfd6r"
Jul 07 15:44:08.550836 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.550745    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4e92cac1-64ea-498d-b27c-30403f6e8b81-konnectivity-ca\") pod \"konnectivity-agent-r9nv8\" (UID: \"4e92cac1-64ea-498d-b27c-30403f6e8b81\") " pod="kube-system/konnectivity-agent-r9nv8"
Jul 07 15:44:08.550836 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.550800    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/69ca53b7-f45f-471b-96b0-1cff9686d44f-cnibin\") pod \"multus-74zmd\" (UID: \"69ca53b7-f45f-471b-96b0-1cff9686d44f\") " pod="openshift-multus/multus-74zmd"
Jul 07 15:44:08.550836 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.550805    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/69ca53b7-f45f-471b-96b0-1cff9686d44f-host-run-multus-certs\") pod \"multus-74zmd\" (UID: \"69ca53b7-f45f-471b-96b0-1cff9686d44f\") " pod="openshift-multus/multus-74zmd"
Jul 07 15:44:08.551240 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.549704    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/69ca53b7-f45f-471b-96b0-1cff9686d44f-host-var-lib-kubelet\") pod \"multus-74zmd\" (UID: \"69ca53b7-f45f-471b-96b0-1cff9686d44f\") " pod="openshift-multus/multus-74zmd"
Jul 07 15:44:08.551240 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.551043    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb-ovnkube-config\") pod \"ovnkube-node-vfd6r\" (UID: \"b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfd6r"
Jul 07 15:44:08.551240 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.551069    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0e3de181-9d7a-4f84-ae08-d018e710109c-cni-binary-copy\") pod \"multus-additional-cni-plugins-zbxwc\" (UID: \"0e3de181-9d7a-4f84-ae08-d018e710109c\") " pod="openshift-multus/multus-additional-cni-plugins-zbxwc"
Jul 07 15:44:08.553364 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.553338    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb-ovn-node-metrics-cert\") pod \"ovnkube-node-vfd6r\" (UID: \"b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfd6r"
Jul 07 15:44:08.554841 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.554815    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4e92cac1-64ea-498d-b27c-30403f6e8b81-agent-certs\") pod \"konnectivity-agent-r9nv8\" (UID: \"4e92cac1-64ea-498d-b27c-30403f6e8b81\") " pod="kube-system/konnectivity-agent-r9nv8"
Jul 07 15:44:08.556241 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.556216    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fczmt\" (UniqueName: \"kubernetes.io/projected/ff08ea94-65cb-4ca1-a272-3de7627d10b8-kube-api-access-fczmt\") pod \"node-ca-shpjq\" (UID: \"ff08ea94-65cb-4ca1-a272-3de7627d10b8\") " pod="openshift-image-registry/node-ca-shpjq"
Jul 07 15:44:08.557160 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.557135    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wvhw\" (UniqueName: \"kubernetes.io/projected/b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb-kube-api-access-7wvhw\") pod \"ovnkube-node-vfd6r\" (UID: \"b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfd6r"
Jul 07 15:44:08.557260 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.557140    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf2t6\" (UniqueName: \"kubernetes.io/projected/b9e1b70f-f064-49bb-ab99-39d7bb57a5d1-kube-api-access-kf2t6\") pod \"aws-ebs-csi-driver-node-lq9gt\" (UID: \"b9e1b70f-f064-49bb-ab99-39d7bb57a5d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lq9gt"
Jul 07 15:44:08.557388 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.557363    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmgcl\" (UniqueName: \"kubernetes.io/projected/06e24adc-93cb-495b-94f2-d1435ae35b39-kube-api-access-mmgcl\") pod \"iptables-alerter-v7lzm\" (UID: \"06e24adc-93cb-495b-94f2-d1435ae35b39\") " pod="openshift-network-operator/iptables-alerter-v7lzm"
Jul 07 15:44:08.559844 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.559821    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dt5k4\" (UniqueName: \"kubernetes.io/projected/0e3de181-9d7a-4f84-ae08-d018e710109c-kube-api-access-dt5k4\") pod \"multus-additional-cni-plugins-zbxwc\" (UID: \"0e3de181-9d7a-4f84-ae08-d018e710109c\") " pod="openshift-multus/multus-additional-cni-plugins-zbxwc"
Jul 07 15:44:08.560183 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.560163    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqsvf\" (UniqueName: \"kubernetes.io/projected/22f1cba0-dd6e-4f5a-8d37-d83d1f938ace-kube-api-access-bqsvf\") pod \"node-resolver-6cgnn\" (UID: \"22f1cba0-dd6e-4f5a-8d37-d83d1f938ace\") " pod="openshift-dns/node-resolver-6cgnn"
Jul 07 15:44:08.560659 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.560636    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79sx2\" (UniqueName: \"kubernetes.io/projected/69ca53b7-f45f-471b-96b0-1cff9686d44f-kube-api-access-79sx2\") pod \"multus-74zmd\" (UID: \"69ca53b7-f45f-471b-96b0-1cff9686d44f\") " pod="openshift-multus/multus-74zmd"
Jul 07 15:44:08.643518 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.643433    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bfgft"
Jul 07 15:44:08.653973 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.653934    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6cgnn"
Jul 07 15:44:08.663996 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.663965    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-shpjq"
Jul 07 15:44:08.672770 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.672736    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-v7lzm"
Jul 07 15:44:08.681933 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.681909    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vfd6r"
Jul 07 15:44:08.691010 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.690989    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-r9nv8"
Jul 07 15:44:08.697961 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.697925    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lq9gt"
Jul 07 15:44:08.711215 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.710973    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-74zmd"
Jul 07 15:44:08.716994 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.716973    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zbxwc"
Jul 07 15:44:08.953039 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.953000    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a773f68-4a73-415d-abc9-3672aa14a254-metrics-certs\") pod \"network-metrics-daemon-zbhgc\" (UID: \"2a773f68-4a73-415d-abc9-3672aa14a254\") " pod="openshift-multus/network-metrics-daemon-zbhgc"
Jul 07 15:44:08.953237 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:08.953050    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f0d07c9b-7fac-4665-b397-d996f6781c50-original-pull-secret\") pod \"global-pull-secret-syncer-fvkrg\" (UID: \"f0d07c9b-7fac-4665-b397-d996f6781c50\") " pod="kube-system/global-pull-secret-syncer-fvkrg"
Jul 07 15:44:08.953317 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:08.953247    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 15:44:08.953317 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:08.953307    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 15:44:08.953420 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:08.953313    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0d07c9b-7fac-4665-b397-d996f6781c50-original-pull-secret podName:f0d07c9b-7fac-4665-b397-d996f6781c50 nodeName:}" failed. No retries permitted until 2026-07-07 15:44:09.953293378 +0000 UTC m=+4.041719049 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f0d07c9b-7fac-4665-b397-d996f6781c50-original-pull-secret") pod "global-pull-secret-syncer-fvkrg" (UID: "f0d07c9b-7fac-4665-b397-d996f6781c50") : object "kube-system"/"original-pull-secret" not registered
Jul 07 15:44:08.953420 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:08.953356    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a773f68-4a73-415d-abc9-3672aa14a254-metrics-certs podName:2a773f68-4a73-415d-abc9-3672aa14a254 nodeName:}" failed. No retries permitted until 2026-07-07 15:44:09.953343703 +0000 UTC m=+4.041769360 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2a773f68-4a73-415d-abc9-3672aa14a254-metrics-certs") pod "network-metrics-daemon-zbhgc" (UID: "2a773f68-4a73-415d-abc9-3672aa14a254") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 15:44:09.053732 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:09.053694    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4wkf\" (UniqueName: \"kubernetes.io/projected/e86522b9-3dc8-41c0-8d8a-40229b4795a7-kube-api-access-m4wkf\") pod \"network-check-target-tx5v9\" (UID: \"e86522b9-3dc8-41c0-8d8a-40229b4795a7\") " pod="openshift-network-diagnostics/network-check-target-tx5v9"
Jul 07 15:44:09.053923 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:09.053844    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 15:44:09.053923 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:09.053866    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 15:44:09.053923 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:09.053876    2566 projected.go:196] Error preparing data for projected volume kube-api-access-m4wkf for pod openshift-network-diagnostics/network-check-target-tx5v9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 15:44:09.054072 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:09.053950    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e86522b9-3dc8-41c0-8d8a-40229b4795a7-kube-api-access-m4wkf podName:e86522b9-3dc8-41c0-8d8a-40229b4795a7 nodeName:}" failed. No retries permitted until 2026-07-07 15:44:10.053914938 +0000 UTC m=+4.142340589 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-m4wkf" (UniqueName: "kubernetes.io/projected/e86522b9-3dc8-41c0-8d8a-40229b4795a7-kube-api-access-m4wkf") pod "network-check-target-tx5v9" (UID: "e86522b9-3dc8-41c0-8d8a-40229b4795a7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 15:44:09.360218 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:09.359943    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e18ef02_9812_4ed6_828e_3f88d2429a7d.slice/crio-96a6ffc23716d3deabaa64469549c4cbd5781f8734ff4a7b713da055c3074c61 WatchSource:0}: Error finding container 96a6ffc23716d3deabaa64469549c4cbd5781f8734ff4a7b713da055c3074c61: Status 404 returned error can't find the container with id 96a6ffc23716d3deabaa64469549c4cbd5781f8734ff4a7b713da055c3074c61
Jul 07 15:44:09.363065 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:09.363039    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e3de181_9d7a_4f84_ae08_d018e710109c.slice/crio-41f18988a3b114c917af0cce0dfa06482ce335ecc8e5b0a5121919083e1f8e68 WatchSource:0}: Error finding container 41f18988a3b114c917af0cce0dfa06482ce335ecc8e5b0a5121919083e1f8e68: Status 404 returned error can't find the container with id 41f18988a3b114c917af0cce0dfa06482ce335ecc8e5b0a5121919083e1f8e68
Jul 07 15:44:09.365519 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:09.365481    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e92cac1_64ea_498d_b27c_30403f6e8b81.slice/crio-b6435c526cfa638a98314f8dea9a6266afbbac2829dac44124d1ee457dbcf3b8 WatchSource:0}: Error finding container b6435c526cfa638a98314f8dea9a6266afbbac2829dac44124d1ee457dbcf3b8: Status 404 returned error can't find the container with id b6435c526cfa638a98314f8dea9a6266afbbac2829dac44124d1ee457dbcf3b8
Jul 07 15:44:09.366804 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:09.366780    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06e24adc_93cb_495b_94f2_d1435ae35b39.slice/crio-2a834404b6a729ccb63649cf4933429404e374154c92b16e449e7f00d299cbaf WatchSource:0}: Error finding container 2a834404b6a729ccb63649cf4933429404e374154c92b16e449e7f00d299cbaf: Status 404 returned error can't find the container with id 2a834404b6a729ccb63649cf4933429404e374154c92b16e449e7f00d299cbaf
Jul 07 15:44:09.385940 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:09.385908    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 15:39:08 +0000 UTC" deadline="2028-02-05 08:57:43.901576097 +0000 UTC"
Jul 07 15:44:09.385940 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:09.385937    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13865h13m34.515642987s"
Jul 07 15:44:09.391246 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:09.391218    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff08ea94_65cb_4ca1_a272_3de7627d10b8.slice/crio-e40d7733893c4fdaa2f5eb4c09d8756e21f8eb32c49794b48d5e5b8de98fd75e WatchSource:0}: Error finding container e40d7733893c4fdaa2f5eb4c09d8756e21f8eb32c49794b48d5e5b8de98fd75e: Status 404 returned error can't find the container with id e40d7733893c4fdaa2f5eb4c09d8756e21f8eb32c49794b48d5e5b8de98fd75e
Jul 07 15:44:09.392508 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:09.392245    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9407f8f_3dbe_4dd8_8c25_dd8bab50dbfb.slice/crio-581659ccfe2d55e86552919fac4693d062ed1fdc34ab452f0727c022b0c2ce81 WatchSource:0}: Error finding container 581659ccfe2d55e86552919fac4693d062ed1fdc34ab452f0727c022b0c2ce81: Status 404 returned error can't find the container with id 581659ccfe2d55e86552919fac4693d062ed1fdc34ab452f0727c022b0c2ce81
Jul 07 15:44:09.393217 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:09.393082    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22f1cba0_dd6e_4f5a_8d37_d83d1f938ace.slice/crio-b04aa8a13fe34d865abc785daa1b1006f46db65df1cd0487937e0946d325d019 WatchSource:0}: Error finding container b04aa8a13fe34d865abc785daa1b1006f46db65df1cd0487937e0946d325d019: Status 404 returned error can't find the container with id b04aa8a13fe34d865abc785daa1b1006f46db65df1cd0487937e0946d325d019
Jul 07 15:44:09.394420 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:09.394362    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69ca53b7_f45f_471b_96b0_1cff9686d44f.slice/crio-b1d0839146a5f69e16e6e570df6d04ed7add4c8e6b8fe70390b457daeeeca654 WatchSource:0}: Error finding container b1d0839146a5f69e16e6e570df6d04ed7add4c8e6b8fe70390b457daeeeca654: Status 404 returned error can't find the container with id b1d0839146a5f69e16e6e570df6d04ed7add4c8e6b8fe70390b457daeeeca654
Jul 07 15:44:09.468694 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:09.468672    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zbhgc"
Jul 07 15:44:09.468818 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:09.468790    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zbhgc" podUID="2a773f68-4a73-415d-abc9-3672aa14a254"
Jul 07 15:44:09.477702 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:09.477675    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6cgnn" event={"ID":"22f1cba0-dd6e-4f5a-8d37-d83d1f938ace","Type":"ContainerStarted","Data":"b04aa8a13fe34d865abc785daa1b1006f46db65df1cd0487937e0946d325d019"}
Jul 07 15:44:09.478936 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:09.478914    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vfd6r" event={"ID":"b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb","Type":"ContainerStarted","Data":"581659ccfe2d55e86552919fac4693d062ed1fdc34ab452f0727c022b0c2ce81"}
Jul 07 15:44:09.480736 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:09.480708    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zbxwc" event={"ID":"0e3de181-9d7a-4f84-ae08-d018e710109c","Type":"ContainerStarted","Data":"41f18988a3b114c917af0cce0dfa06482ce335ecc8e5b0a5121919083e1f8e68"}
Jul 07 15:44:09.482870 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:09.482847    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bfgft" event={"ID":"6e18ef02-9812-4ed6-828e-3f88d2429a7d","Type":"ContainerStarted","Data":"96a6ffc23716d3deabaa64469549c4cbd5781f8734ff4a7b713da055c3074c61"}
Jul 07 15:44:09.483838 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:09.483816    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lq9gt" event={"ID":"b9e1b70f-f064-49bb-ab99-39d7bb57a5d1","Type":"ContainerStarted","Data":"f9606af1cea3549149ae8589bb8a325700fbbde33aa179b8dfb2f865b97a938c"}
Jul 07 15:44:09.486689 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:09.486670    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-74zmd" event={"ID":"69ca53b7-f45f-471b-96b0-1cff9686d44f","Type":"ContainerStarted","Data":"b1d0839146a5f69e16e6e570df6d04ed7add4c8e6b8fe70390b457daeeeca654"}
Jul 07 15:44:09.487635 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:09.487612    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-shpjq" event={"ID":"ff08ea94-65cb-4ca1-a272-3de7627d10b8","Type":"ContainerStarted","Data":"e40d7733893c4fdaa2f5eb4c09d8756e21f8eb32c49794b48d5e5b8de98fd75e"}
Jul 07 15:44:09.488603 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:09.488581    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-v7lzm" event={"ID":"06e24adc-93cb-495b-94f2-d1435ae35b39","Type":"ContainerStarted","Data":"2a834404b6a729ccb63649cf4933429404e374154c92b16e449e7f00d299cbaf"}
Jul 07 15:44:09.489589 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:09.489570    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-r9nv8" event={"ID":"4e92cac1-64ea-498d-b27c-30403f6e8b81","Type":"ContainerStarted","Data":"b6435c526cfa638a98314f8dea9a6266afbbac2829dac44124d1ee457dbcf3b8"}
Jul 07 15:44:09.961387 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:09.961201    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a773f68-4a73-415d-abc9-3672aa14a254-metrics-certs\") pod \"network-metrics-daemon-zbhgc\" (UID: \"2a773f68-4a73-415d-abc9-3672aa14a254\") " pod="openshift-multus/network-metrics-daemon-zbhgc"
Jul 07 15:44:09.961563 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:09.961405    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f0d07c9b-7fac-4665-b397-d996f6781c50-original-pull-secret\") pod \"global-pull-secret-syncer-fvkrg\" (UID: \"f0d07c9b-7fac-4665-b397-d996f6781c50\") " pod="kube-system/global-pull-secret-syncer-fvkrg"
Jul 07 15:44:09.961563 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:09.961363    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 15:44:09.961563 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:09.961515    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 15:44:09.961563 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:09.961527    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a773f68-4a73-415d-abc9-3672aa14a254-metrics-certs podName:2a773f68-4a73-415d-abc9-3672aa14a254 nodeName:}" failed. No retries permitted until 2026-07-07 15:44:11.961511643 +0000 UTC m=+6.049937292 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2a773f68-4a73-415d-abc9-3672aa14a254-metrics-certs") pod "network-metrics-daemon-zbhgc" (UID: "2a773f68-4a73-415d-abc9-3672aa14a254") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 15:44:09.961563 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:09.961567    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0d07c9b-7fac-4665-b397-d996f6781c50-original-pull-secret podName:f0d07c9b-7fac-4665-b397-d996f6781c50 nodeName:}" failed. No retries permitted until 2026-07-07 15:44:11.961559652 +0000 UTC m=+6.049985301 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f0d07c9b-7fac-4665-b397-d996f6781c50-original-pull-secret") pod "global-pull-secret-syncer-fvkrg" (UID: "f0d07c9b-7fac-4665-b397-d996f6781c50") : object "kube-system"/"original-pull-secret" not registered
Jul 07 15:44:10.062333 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:10.062291    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4wkf\" (UniqueName: \"kubernetes.io/projected/e86522b9-3dc8-41c0-8d8a-40229b4795a7-kube-api-access-m4wkf\") pod \"network-check-target-tx5v9\" (UID: \"e86522b9-3dc8-41c0-8d8a-40229b4795a7\") " pod="openshift-network-diagnostics/network-check-target-tx5v9"
Jul 07 15:44:10.062527 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:10.062489    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 15:44:10.062527 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:10.062517    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 15:44:10.062640 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:10.062532    2566 projected.go:196] Error preparing data for projected volume kube-api-access-m4wkf for pod openshift-network-diagnostics/network-check-target-tx5v9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 15:44:10.062640 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:10.062595    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e86522b9-3dc8-41c0-8d8a-40229b4795a7-kube-api-access-m4wkf podName:e86522b9-3dc8-41c0-8d8a-40229b4795a7 nodeName:}" failed. No retries permitted until 2026-07-07 15:44:12.06257468 +0000 UTC m=+6.151000351 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-m4wkf" (UniqueName: "kubernetes.io/projected/e86522b9-3dc8-41c0-8d8a-40229b4795a7-kube-api-access-m4wkf") pod "network-check-target-tx5v9" (UID: "e86522b9-3dc8-41c0-8d8a-40229b4795a7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 15:44:10.386546 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:10.386416    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 15:39:08 +0000 UTC" deadline="2028-02-27 20:55:18.472485474 +0000 UTC"
Jul 07 15:44:10.386546 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:10.386462    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14405h11m8.086028682s"
Jul 07 15:44:10.471246 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:10.471208    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fvkrg"
Jul 07 15:44:10.471389 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:10.471358    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fvkrg" podUID="f0d07c9b-7fac-4665-b397-d996f6781c50"
Jul 07 15:44:10.472075 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:10.472053    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tx5v9"
Jul 07 15:44:10.472202 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:10.472168    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tx5v9" podUID="e86522b9-3dc8-41c0-8d8a-40229b4795a7"
Jul 07 15:44:10.509809 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:10.509185    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-101.ec2.internal" event={"ID":"9b6c8b6d42e5b52ab4cf9f706a9be0f7","Type":"ContainerStarted","Data":"33159ef37dc749949876fdd24e550111a5078cc2bab2ad287e2aab75f3f2708c"}
Jul 07 15:44:10.517980 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:10.517398    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-101.ec2.internal" event={"ID":"a9ccd326e5056db2ec15b8f22d5755cf","Type":"ContainerStarted","Data":"9fe7752307afda25ebeeb2b57ca4e590c20dd722a06c0f2b705c1f25321982b6"}
Jul 07 15:44:10.536913 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:10.536079    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-101.ec2.internal" podStartSLOduration=3.536059671 podStartE2EDuration="3.536059671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 15:44:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:44:09.505380837 +0000 UTC m=+3.593806512" watchObservedRunningTime="2026-07-07 15:44:10.536059671 +0000 UTC m=+4.624485345"
Jul 07 15:44:11.468768 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:11.468730    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zbhgc"
Jul 07 15:44:11.469325 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:11.468880    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zbhgc" podUID="2a773f68-4a73-415d-abc9-3672aa14a254"
Jul 07 15:44:11.534836 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:11.534795    2566 generic.go:356] "Generic (PLEG): container finished" podID="a9ccd326e5056db2ec15b8f22d5755cf" containerID="9fe7752307afda25ebeeb2b57ca4e590c20dd722a06c0f2b705c1f25321982b6" exitCode=0
Jul 07 15:44:11.535090 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:11.534869    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-101.ec2.internal" event={"ID":"a9ccd326e5056db2ec15b8f22d5755cf","Type":"ContainerDied","Data":"9fe7752307afda25ebeeb2b57ca4e590c20dd722a06c0f2b705c1f25321982b6"}
Jul 07 15:44:11.977947 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:11.977173    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a773f68-4a73-415d-abc9-3672aa14a254-metrics-certs\") pod \"network-metrics-daemon-zbhgc\" (UID: \"2a773f68-4a73-415d-abc9-3672aa14a254\") " pod="openshift-multus/network-metrics-daemon-zbhgc"
Jul 07 15:44:11.977947 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:11.977225    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f0d07c9b-7fac-4665-b397-d996f6781c50-original-pull-secret\") pod \"global-pull-secret-syncer-fvkrg\" (UID: \"f0d07c9b-7fac-4665-b397-d996f6781c50\") " pod="kube-system/global-pull-secret-syncer-fvkrg"
Jul 07 15:44:11.977947 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:11.977392    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 15:44:11.977947 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:11.977457    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0d07c9b-7fac-4665-b397-d996f6781c50-original-pull-secret podName:f0d07c9b-7fac-4665-b397-d996f6781c50 nodeName:}" failed. No retries permitted until 2026-07-07 15:44:15.977438606 +0000 UTC m=+10.065864262 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f0d07c9b-7fac-4665-b397-d996f6781c50-original-pull-secret") pod "global-pull-secret-syncer-fvkrg" (UID: "f0d07c9b-7fac-4665-b397-d996f6781c50") : object "kube-system"/"original-pull-secret" not registered
Jul 07 15:44:11.977947 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:11.977861    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 15:44:11.977947 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:11.977913    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a773f68-4a73-415d-abc9-3672aa14a254-metrics-certs podName:2a773f68-4a73-415d-abc9-3672aa14a254 nodeName:}" failed. No retries permitted until 2026-07-07 15:44:15.977896543 +0000 UTC m=+10.066322215 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2a773f68-4a73-415d-abc9-3672aa14a254-metrics-certs") pod "network-metrics-daemon-zbhgc" (UID: "2a773f68-4a73-415d-abc9-3672aa14a254") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 15:44:12.080423 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:12.079606    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4wkf\" (UniqueName: \"kubernetes.io/projected/e86522b9-3dc8-41c0-8d8a-40229b4795a7-kube-api-access-m4wkf\") pod \"network-check-target-tx5v9\" (UID: \"e86522b9-3dc8-41c0-8d8a-40229b4795a7\") " pod="openshift-network-diagnostics/network-check-target-tx5v9"
Jul 07 15:44:12.080423 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:12.079810    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 15:44:12.080423 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:12.079832    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 15:44:12.080423 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:12.079846    2566 projected.go:196] Error preparing data for projected volume kube-api-access-m4wkf for pod openshift-network-diagnostics/network-check-target-tx5v9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 15:44:12.080423 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:12.079908    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e86522b9-3dc8-41c0-8d8a-40229b4795a7-kube-api-access-m4wkf podName:e86522b9-3dc8-41c0-8d8a-40229b4795a7 nodeName:}" failed. No retries permitted until 2026-07-07 15:44:16.079887856 +0000 UTC m=+10.168313506 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-m4wkf" (UniqueName: "kubernetes.io/projected/e86522b9-3dc8-41c0-8d8a-40229b4795a7-kube-api-access-m4wkf") pod "network-check-target-tx5v9" (UID: "e86522b9-3dc8-41c0-8d8a-40229b4795a7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 15:44:12.470632 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:12.470088    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tx5v9"
Jul 07 15:44:12.470632 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:12.470223    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tx5v9" podUID="e86522b9-3dc8-41c0-8d8a-40229b4795a7"
Jul 07 15:44:12.471479 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:12.471289    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fvkrg"
Jul 07 15:44:12.471479 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:12.471406    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fvkrg" podUID="f0d07c9b-7fac-4665-b397-d996f6781c50"
Jul 07 15:44:12.569993 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:12.569389    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-101.ec2.internal" event={"ID":"a9ccd326e5056db2ec15b8f22d5755cf","Type":"ContainerStarted","Data":"04603a65d6f91bf52a110d36c37284d1a69e48a4bdb618df4c6869f6e4ac214a"}
Jul 07 15:44:13.468544 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:13.468512    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zbhgc"
Jul 07 15:44:13.468742 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:13.468668    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zbhgc" podUID="2a773f68-4a73-415d-abc9-3672aa14a254"
Jul 07 15:44:14.470992 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:14.470264    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fvkrg"
Jul 07 15:44:14.470992 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:14.470469    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fvkrg" podUID="f0d07c9b-7fac-4665-b397-d996f6781c50"
Jul 07 15:44:14.470992 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:14.470822    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tx5v9"
Jul 07 15:44:14.470992 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:14.470952    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tx5v9" podUID="e86522b9-3dc8-41c0-8d8a-40229b4795a7"
Jul 07 15:44:15.469148 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:15.469110    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zbhgc"
Jul 07 15:44:15.469318 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:15.469263    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zbhgc" podUID="2a773f68-4a73-415d-abc9-3672aa14a254"
Jul 07 15:44:16.014976 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:16.014935    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a773f68-4a73-415d-abc9-3672aa14a254-metrics-certs\") pod \"network-metrics-daemon-zbhgc\" (UID: \"2a773f68-4a73-415d-abc9-3672aa14a254\") " pod="openshift-multus/network-metrics-daemon-zbhgc"
Jul 07 15:44:16.015512 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:16.014983    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f0d07c9b-7fac-4665-b397-d996f6781c50-original-pull-secret\") pod \"global-pull-secret-syncer-fvkrg\" (UID: \"f0d07c9b-7fac-4665-b397-d996f6781c50\") " pod="kube-system/global-pull-secret-syncer-fvkrg"
Jul 07 15:44:16.015512 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:16.015090    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 15:44:16.015512 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:16.015136    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 15:44:16.015512 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:16.015150    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0d07c9b-7fac-4665-b397-d996f6781c50-original-pull-secret podName:f0d07c9b-7fac-4665-b397-d996f6781c50 nodeName:}" failed. No retries permitted until 2026-07-07 15:44:24.015137095 +0000 UTC m=+18.103562744 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f0d07c9b-7fac-4665-b397-d996f6781c50-original-pull-secret") pod "global-pull-secret-syncer-fvkrg" (UID: "f0d07c9b-7fac-4665-b397-d996f6781c50") : object "kube-system"/"original-pull-secret" not registered
Jul 07 15:44:16.015512 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:16.015193    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a773f68-4a73-415d-abc9-3672aa14a254-metrics-certs podName:2a773f68-4a73-415d-abc9-3672aa14a254 nodeName:}" failed. No retries permitted until 2026-07-07 15:44:24.015179081 +0000 UTC m=+18.103604730 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2a773f68-4a73-415d-abc9-3672aa14a254-metrics-certs") pod "network-metrics-daemon-zbhgc" (UID: "2a773f68-4a73-415d-abc9-3672aa14a254") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 15:44:16.115781 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:16.115746    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4wkf\" (UniqueName: \"kubernetes.io/projected/e86522b9-3dc8-41c0-8d8a-40229b4795a7-kube-api-access-m4wkf\") pod \"network-check-target-tx5v9\" (UID: \"e86522b9-3dc8-41c0-8d8a-40229b4795a7\") " pod="openshift-network-diagnostics/network-check-target-tx5v9"
Jul 07 15:44:16.116381 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:16.115966    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 15:44:16.116381 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:16.115992    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 15:44:16.116381 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:16.116004    2566 projected.go:196] Error preparing data for projected volume kube-api-access-m4wkf for pod openshift-network-diagnostics/network-check-target-tx5v9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 15:44:16.116381 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:16.116065    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e86522b9-3dc8-41c0-8d8a-40229b4795a7-kube-api-access-m4wkf podName:e86522b9-3dc8-41c0-8d8a-40229b4795a7 nodeName:}" failed. No retries permitted until 2026-07-07 15:44:24.116047005 +0000 UTC m=+18.204472656 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-m4wkf" (UniqueName: "kubernetes.io/projected/e86522b9-3dc8-41c0-8d8a-40229b4795a7-kube-api-access-m4wkf") pod "network-check-target-tx5v9" (UID: "e86522b9-3dc8-41c0-8d8a-40229b4795a7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 15:44:16.470004 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:16.469734    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tx5v9"
Jul 07 15:44:16.470004 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:16.469854    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tx5v9" podUID="e86522b9-3dc8-41c0-8d8a-40229b4795a7"
Jul 07 15:44:16.470004 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:16.469867    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fvkrg"
Jul 07 15:44:16.470004 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:16.469959    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fvkrg" podUID="f0d07c9b-7fac-4665-b397-d996f6781c50"
Jul 07 15:44:17.468560 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:17.468537    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zbhgc"
Jul 07 15:44:17.468872 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:17.468645    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zbhgc" podUID="2a773f68-4a73-415d-abc9-3672aa14a254"
Jul 07 15:44:18.469303 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:18.469269    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fvkrg"
Jul 07 15:44:18.469303 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:18.469295    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tx5v9"
Jul 07 15:44:18.469782 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:18.469391    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fvkrg" podUID="f0d07c9b-7fac-4665-b397-d996f6781c50"
Jul 07 15:44:18.469782 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:18.469520    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tx5v9" podUID="e86522b9-3dc8-41c0-8d8a-40229b4795a7"
Jul 07 15:44:19.468965 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:19.468933    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zbhgc"
Jul 07 15:44:19.469160 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:19.469068    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zbhgc" podUID="2a773f68-4a73-415d-abc9-3672aa14a254"
Jul 07 15:44:20.468980 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:20.468945    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tx5v9"
Jul 07 15:44:20.469391 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:20.468955    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fvkrg"
Jul 07 15:44:20.469391 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:20.469087    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tx5v9" podUID="e86522b9-3dc8-41c0-8d8a-40229b4795a7"
Jul 07 15:44:20.469391 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:20.469247    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fvkrg" podUID="f0d07c9b-7fac-4665-b397-d996f6781c50"
Jul 07 15:44:20.587612 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:20.587573    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-r9nv8" event={"ID":"4e92cac1-64ea-498d-b27c-30403f6e8b81","Type":"ContainerStarted","Data":"065ea1984a205deceb75776ea029b73c667842e16ed1b52321587f10d94463fd"}
Jul 07 15:44:20.608234 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:20.608179    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-r9nv8" podStartSLOduration=2.894003595 podStartE2EDuration="13.608158871s" totalImagesPullingTime="10.714155276s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:44:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:44:20.60798981 +0000 UTC m=+14.696415482" watchObservedRunningTime="2026-07-07 15:44:20.608158871 +0000 UTC m=+14.696584541"
Jul 07 15:44:20.608396 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:20.608334    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-101.ec2.internal" podStartSLOduration=13.608327576 podStartE2EDuration="13.608327576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 15:44:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:44:12.592671415 +0000 UTC m=+6.681097088" watchObservedRunningTime="2026-07-07 15:44:20.608327576 +0000 UTC m=+14.696753246"
Jul 07 15:44:20.930021 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:20.929942    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-r9nv8"
Jul 07 15:44:20.930576 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:20.930550    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-r9nv8"
Jul 07 15:44:21.469071 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:21.469035    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zbhgc"
Jul 07 15:44:21.469494 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:21.469184    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zbhgc" podUID="2a773f68-4a73-415d-abc9-3672aa14a254"
Jul 07 15:44:22.469263 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:22.469221    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fvkrg"
Jul 07 15:44:22.469725 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:22.469369    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fvkrg" podUID="f0d07c9b-7fac-4665-b397-d996f6781c50"
Jul 07 15:44:22.469725 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:22.469412    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tx5v9"
Jul 07 15:44:22.469725 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:22.469525    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tx5v9" podUID="e86522b9-3dc8-41c0-8d8a-40229b4795a7"
Jul 07 15:44:22.591796 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:22.591772    2566 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 07 15:44:23.468640 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:23.468609    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zbhgc"
Jul 07 15:44:23.468873 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:23.468735    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zbhgc" podUID="2a773f68-4a73-415d-abc9-3672aa14a254"
Jul 07 15:44:24.073987 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:24.073954    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a773f68-4a73-415d-abc9-3672aa14a254-metrics-certs\") pod \"network-metrics-daemon-zbhgc\" (UID: \"2a773f68-4a73-415d-abc9-3672aa14a254\") " pod="openshift-multus/network-metrics-daemon-zbhgc"
Jul 07 15:44:24.073987 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:24.073989    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f0d07c9b-7fac-4665-b397-d996f6781c50-original-pull-secret\") pod \"global-pull-secret-syncer-fvkrg\" (UID: \"f0d07c9b-7fac-4665-b397-d996f6781c50\") " pod="kube-system/global-pull-secret-syncer-fvkrg"
Jul 07 15:44:24.074452 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:24.074117    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 15:44:24.074452 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:24.074184    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a773f68-4a73-415d-abc9-3672aa14a254-metrics-certs podName:2a773f68-4a73-415d-abc9-3672aa14a254 nodeName:}" failed. No retries permitted until 2026-07-07 15:44:40.074166283 +0000 UTC m=+34.162591951 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2a773f68-4a73-415d-abc9-3672aa14a254-metrics-certs") pod "network-metrics-daemon-zbhgc" (UID: "2a773f68-4a73-415d-abc9-3672aa14a254") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 15:44:24.074452 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:24.074117    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 15:44:24.074452 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:24.074220    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0d07c9b-7fac-4665-b397-d996f6781c50-original-pull-secret podName:f0d07c9b-7fac-4665-b397-d996f6781c50 nodeName:}" failed. No retries permitted until 2026-07-07 15:44:40.074212328 +0000 UTC m=+34.162637977 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f0d07c9b-7fac-4665-b397-d996f6781c50-original-pull-secret") pod "global-pull-secret-syncer-fvkrg" (UID: "f0d07c9b-7fac-4665-b397-d996f6781c50") : object "kube-system"/"original-pull-secret" not registered
Jul 07 15:44:24.174325 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:24.174294    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4wkf\" (UniqueName: \"kubernetes.io/projected/e86522b9-3dc8-41c0-8d8a-40229b4795a7-kube-api-access-m4wkf\") pod \"network-check-target-tx5v9\" (UID: \"e86522b9-3dc8-41c0-8d8a-40229b4795a7\") " pod="openshift-network-diagnostics/network-check-target-tx5v9"
Jul 07 15:44:24.174481 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:24.174417    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 15:44:24.174481 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:24.174432    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 15:44:24.174481 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:24.174442    2566 projected.go:196] Error preparing data for projected volume kube-api-access-m4wkf for pod openshift-network-diagnostics/network-check-target-tx5v9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 15:44:24.174587 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:24.174487    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e86522b9-3dc8-41c0-8d8a-40229b4795a7-kube-api-access-m4wkf podName:e86522b9-3dc8-41c0-8d8a-40229b4795a7 nodeName:}" failed. No retries permitted until 2026-07-07 15:44:40.174474773 +0000 UTC m=+34.262900420 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-m4wkf" (UniqueName: "kubernetes.io/projected/e86522b9-3dc8-41c0-8d8a-40229b4795a7-kube-api-access-m4wkf") pod "network-check-target-tx5v9" (UID: "e86522b9-3dc8-41c0-8d8a-40229b4795a7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 15:44:24.449852 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:24.449823    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-r9nv8"
Jul 07 15:44:24.450018 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:24.449954    2566 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 07 15:44:24.450556 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:24.450534    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-r9nv8"
Jul 07 15:44:24.469077 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:24.469035    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fvkrg"
Jul 07 15:44:24.469230 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:24.469165    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fvkrg" podUID="f0d07c9b-7fac-4665-b397-d996f6781c50"
Jul 07 15:44:24.469409 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:24.469389    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tx5v9"
Jul 07 15:44:24.469519 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:24.469495    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tx5v9" podUID="e86522b9-3dc8-41c0-8d8a-40229b4795a7"
Jul 07 15:44:25.469353 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:25.469321    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zbhgc"
Jul 07 15:44:25.469803 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:25.469449    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zbhgc" podUID="2a773f68-4a73-415d-abc9-3672aa14a254"
Jul 07 15:44:26.470423 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:26.470377    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fvkrg"
Jul 07 15:44:26.470863 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:26.470502    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fvkrg" podUID="f0d07c9b-7fac-4665-b397-d996f6781c50"
Jul 07 15:44:26.470863 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:26.470548    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tx5v9"
Jul 07 15:44:26.470863 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:26.470674    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tx5v9" podUID="e86522b9-3dc8-41c0-8d8a-40229b4795a7"
Jul 07 15:44:27.469019 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:27.468984    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zbhgc"
Jul 07 15:44:27.469189 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:27.469130    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zbhgc" podUID="2a773f68-4a73-415d-abc9-3672aa14a254"
Jul 07 15:44:28.468952 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:28.468717    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fvkrg"
Jul 07 15:44:28.469338 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:28.469036    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fvkrg" podUID="f0d07c9b-7fac-4665-b397-d996f6781c50"
Jul 07 15:44:28.469338 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:28.468717    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tx5v9"
Jul 07 15:44:28.469338 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:28.469129    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tx5v9" podUID="e86522b9-3dc8-41c0-8d8a-40229b4795a7"
Jul 07 15:44:28.604822 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:28.604593    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-74zmd" event={"ID":"69ca53b7-f45f-471b-96b0-1cff9686d44f","Type":"ContainerStarted","Data":"a07e62ca867ad78cda960a4f71ae59a6322169c13b8849d6d065a0058dbccdac"}
Jul 07 15:44:28.606095 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:28.606061    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-shpjq" event={"ID":"ff08ea94-65cb-4ca1-a272-3de7627d10b8","Type":"ContainerStarted","Data":"7978e949f9d9aa98062e4c3dd27fdaf2165587100fb12944b7c0b7fc63ebc88e"}
Jul 07 15:44:28.607382 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:28.607354    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6cgnn" event={"ID":"22f1cba0-dd6e-4f5a-8d37-d83d1f938ace","Type":"ContainerStarted","Data":"c772225386e001a548c06ce78a7605764c7e4da988277f6247790db39739baa9"}
Jul 07 15:44:28.608845 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:28.608823    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vfd6r" event={"ID":"b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb","Type":"ContainerStarted","Data":"27704b42a22471e8de4d72ec9a8afd7ecd0254739389d5757d6355f53073befe"}
Jul 07 15:44:28.610235 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:28.610215    2566 generic.go:356] "Generic (PLEG): container finished" podID="0e3de181-9d7a-4f84-ae08-d018e710109c" containerID="45c6d4483466d62ac2429c52078933f9ccf8bfeafa1c2ed4edc2e46afdead5f3" exitCode=0
Jul 07 15:44:28.610321 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:28.610276    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zbxwc" event={"ID":"0e3de181-9d7a-4f84-ae08-d018e710109c","Type":"ContainerDied","Data":"45c6d4483466d62ac2429c52078933f9ccf8bfeafa1c2ed4edc2e46afdead5f3"}
Jul 07 15:44:28.611821 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:28.611796    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bfgft" event={"ID":"6e18ef02-9812-4ed6-828e-3f88d2429a7d","Type":"ContainerStarted","Data":"e4b72d109468d39b0ad6a901cf7d02ba32b2cd17b28ace040ed5a703b80f2cef"}
Jul 07 15:44:28.618078 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:28.617227    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lq9gt" event={"ID":"b9e1b70f-f064-49bb-ab99-39d7bb57a5d1","Type":"ContainerStarted","Data":"20a4dd4a4655a822049d6851bf352d75c96adb6d6fc7553639a1908ef49c49ad"}
Jul 07 15:44:28.645781 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:28.645740    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-74zmd" podStartSLOduration=2.686587146 podStartE2EDuration="21.64572713s" totalImagesPullingTime="18.959139984s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 15:44:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:44:28.645352835 +0000 UTC m=+22.733778604" watchObservedRunningTime="2026-07-07 15:44:28.64572713 +0000 UTC m=+22.734152797"
Jul 07 15:44:28.668651 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:28.668598    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-bfgft" podStartSLOduration=2.729601964 podStartE2EDuration="21.668576728s" totalImagesPullingTime="18.938974764s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 15:44:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:44:28.6678802 +0000 UTC m=+22.756305870" watchObservedRunningTime="2026-07-07 15:44:28.668576728 +0000 UTC m=+22.757002397"
Jul 07 15:44:28.688406 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:28.688364    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-6cgnn" podStartSLOduration=10.950123285 podStartE2EDuration="21.688346449s" totalImagesPullingTime="10.738223164s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 15:44:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:44:28.688004109 +0000 UTC m=+22.776429781" watchObservedRunningTime="2026-07-07 15:44:28.688346449 +0000 UTC m=+22.776772119"
Jul 07 15:44:28.730367 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:28.730325    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-shpjq" podStartSLOduration=10.991074074 podStartE2EDuration="21.730309192s" totalImagesPullingTime="10.739235118s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 15:44:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:44:28.704877812 +0000 UTC m=+22.793303480" watchObservedRunningTime="2026-07-07 15:44:28.730309192 +0000 UTC m=+22.818734860"
Jul 07 15:44:29.468820 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:29.468789    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zbhgc"
Jul 07 15:44:29.469164 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:29.468969    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zbhgc" podUID="2a773f68-4a73-415d-abc9-3672aa14a254"
Jul 07 15:44:29.622612 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:29.622577    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vfd6r" event={"ID":"b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb","Type":"ContainerStarted","Data":"df44d4e9aafbbf1fc72cdfb522c215c38f7331688eb48a09fa5ee7be14250003"}
Jul 07 15:44:29.622799 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:29.622652    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vfd6r" event={"ID":"b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb","Type":"ContainerStarted","Data":"20ae94d9ed7b7e32c45f4f735af5a518bcd4f70aab28d7e81611cfed1709109f"}
Jul 07 15:44:29.622799 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:29.622682    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vfd6r" event={"ID":"b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb","Type":"ContainerStarted","Data":"2dc08ceed9ecb5e10323f9f7d7cfab6e04b6d850cab6119253efab455431ee5d"}
Jul 07 15:44:29.622799 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:29.622723    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vfd6r" event={"ID":"b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb","Type":"ContainerStarted","Data":"047448cdeabce06cb2d023640a0ff3e37db2b23c3064577bb96a8fa29bc149f2"}
Jul 07 15:44:29.622799 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:29.622755    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vfd6r" event={"ID":"b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb","Type":"ContainerStarted","Data":"7799244d37fe164b944a28b6ffd59f4745626a93ecb5e07643c0f3f1e78c0904"}
Jul 07 15:44:29.624134 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:29.624098    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-v7lzm" event={"ID":"06e24adc-93cb-495b-94f2-d1435ae35b39","Type":"ContainerStarted","Data":"5cce800b80ed48a25e7285714d53c131ca9e0b29ddb6d7acf8e9f3a7ecc68899"}
Jul 07 15:44:29.649835 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:29.649792    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-v7lzm" podStartSLOduration=11.908324613 podStartE2EDuration="22.649777694s" totalImagesPullingTime="10.741453081s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 15:44:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:44:29.64914778 +0000 UTC m=+23.737573448" watchObservedRunningTime="2026-07-07 15:44:29.649777694 +0000 UTC m=+23.738203362"
Jul 07 15:44:30.427527 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:30.427363    2566 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 07 15:44:30.468825 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:30.468795    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tx5v9"
Jul 07 15:44:30.468984 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:30.468933    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tx5v9" podUID="e86522b9-3dc8-41c0-8d8a-40229b4795a7"
Jul 07 15:44:30.469046 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:30.468993    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fvkrg"
Jul 07 15:44:30.469121 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:30.469102    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fvkrg" podUID="f0d07c9b-7fac-4665-b397-d996f6781c50"
Jul 07 15:44:30.627837 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:30.627766    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lq9gt" event={"ID":"b9e1b70f-f064-49bb-ab99-39d7bb57a5d1","Type":"ContainerStarted","Data":"3e8a950239d4ac098956fb768e3baf51495078e14105c88f32cf999c294a2cc6"}
Jul 07 15:44:31.405661 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:31.405543    2566 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-07T15:44:30.42738392Z","UUID":"caab7781-683d-4265-a875-d475e4d15b6b","Handler":null,"Name":"","Endpoint":""}
Jul 07 15:44:31.406993 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:31.406976    2566 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 07 15:44:31.407086 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:31.407023    2566 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 07 15:44:31.468473 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:31.468447    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zbhgc"
Jul 07 15:44:31.468634 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:31.468569    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zbhgc" podUID="2a773f68-4a73-415d-abc9-3672aa14a254"
Jul 07 15:44:31.632509 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:31.632480    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vfd6r" event={"ID":"b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb","Type":"ContainerStarted","Data":"dae9e5abf442060784231d6254036a62a5ed603fa827dc772f2d465fe96a9092"}
Jul 07 15:44:32.468749 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:32.468552    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tx5v9"
Jul 07 15:44:32.468943 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:32.468610    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fvkrg"
Jul 07 15:44:32.468943 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:32.468850    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tx5v9" podUID="e86522b9-3dc8-41c0-8d8a-40229b4795a7"
Jul 07 15:44:32.469065 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:32.468948    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fvkrg" podUID="f0d07c9b-7fac-4665-b397-d996f6781c50"
Jul 07 15:44:32.636789 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:32.636749    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lq9gt" event={"ID":"b9e1b70f-f064-49bb-ab99-39d7bb57a5d1","Type":"ContainerStarted","Data":"1d75af22bfd006b6651e7c1387fe3246285a98c17fb6f8452dad0ed2eb34f400"}
Jul 07 15:44:33.468657 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:33.468557    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zbhgc"
Jul 07 15:44:33.468803 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:33.468715    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zbhgc" podUID="2a773f68-4a73-415d-abc9-3672aa14a254"
Jul 07 15:44:34.469271 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:34.469102    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tx5v9"
Jul 07 15:44:34.469926 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:34.469170    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fvkrg"
Jul 07 15:44:34.469926 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:34.469349    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tx5v9" podUID="e86522b9-3dc8-41c0-8d8a-40229b4795a7"
Jul 07 15:44:34.469926 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:34.469451    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fvkrg" podUID="f0d07c9b-7fac-4665-b397-d996f6781c50"
Jul 07 15:44:34.644407 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:34.644369    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vfd6r" event={"ID":"b9407f8f-3dbe-4dd8-8c25-dd8bab50dbfb","Type":"ContainerStarted","Data":"85be70a6eb9be81fb854ae97eeb9c37d11d657299b1051d723f1eeb55d8c20ce"}
Jul 07 15:44:34.644610 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:34.644592    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vfd6r"
Jul 07 15:44:34.646965 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:34.646473    2566 generic.go:356] "Generic (PLEG): container finished" podID="0e3de181-9d7a-4f84-ae08-d018e710109c" containerID="b90028d05a3128f0d24654b0b49e65da70e8a7b9410174e26455896f63ac6563" exitCode=0
Jul 07 15:44:34.646965 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:34.646507    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zbxwc" event={"ID":"0e3de181-9d7a-4f84-ae08-d018e710109c","Type":"ContainerDied","Data":"b90028d05a3128f0d24654b0b49e65da70e8a7b9410174e26455896f63ac6563"}
Jul 07 15:44:34.661038 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:34.661013    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vfd6r"
Jul 07 15:44:34.689333 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:34.689281    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lq9gt" podStartSLOduration=13.796749063 podStartE2EDuration="27.689265573s" totalImagesPullingTime="13.89251651s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 15:44:07 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:44:32.659052757 +0000 UTC m=+26.747478427" watchObservedRunningTime="2026-07-07 15:44:34.689265573 +0000 UTC m=+28.777691242"
Jul 07 15:44:34.689877 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:34.689846    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-vfd6r" podStartSLOduration=8.713172421 podStartE2EDuration="27.689833355s" totalImagesPullingTime="18.976660934s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 15:44:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:44:34.689196744 +0000 UTC m=+28.777622413" watchObservedRunningTime="2026-07-07 15:44:34.689833355 +0000 UTC m=+28.778259023"
Jul 07 15:44:35.469316 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:35.469276    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zbhgc"
Jul 07 15:44:35.469728 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:35.469434    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zbhgc" podUID="2a773f68-4a73-415d-abc9-3672aa14a254"
Jul 07 15:44:35.649396 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:35.649367    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vfd6r"
Jul 07 15:44:35.649565 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:35.649449    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vfd6r"
Jul 07 15:44:35.664252 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:35.664230    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vfd6r"
Jul 07 15:44:35.851585 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:35.851556    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-fvkrg"]
Jul 07 15:44:35.851714 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:35.851672    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fvkrg"
Jul 07 15:44:35.851771 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:35.851754    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fvkrg" podUID="f0d07c9b-7fac-4665-b397-d996f6781c50"
Jul 07 15:44:35.858975 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:35.858950    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zbhgc"]
Jul 07 15:44:35.859081 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:35.859037    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zbhgc"
Jul 07 15:44:35.859139 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:35.859119    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zbhgc" podUID="2a773f68-4a73-415d-abc9-3672aa14a254"
Jul 07 15:44:35.877857 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:35.877832    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tx5v9"]
Jul 07 15:44:35.877981 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:35.877967    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tx5v9"
Jul 07 15:44:35.878102 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:35.878076    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tx5v9" podUID="e86522b9-3dc8-41c0-8d8a-40229b4795a7"
Jul 07 15:44:36.652981 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:36.652785    2566 generic.go:356] "Generic (PLEG): container finished" podID="0e3de181-9d7a-4f84-ae08-d018e710109c" containerID="08ed550736e8ae1436e87fff7a5d9c98c62124b9d3c830a28f1f256f44db9e34" exitCode=0
Jul 07 15:44:36.652981 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:36.652868    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zbxwc" event={"ID":"0e3de181-9d7a-4f84-ae08-d018e710109c","Type":"ContainerDied","Data":"08ed550736e8ae1436e87fff7a5d9c98c62124b9d3c830a28f1f256f44db9e34"}
Jul 07 15:44:37.468512 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:37.468487    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fvkrg"
Jul 07 15:44:37.468691 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:37.468592    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zbhgc"
Jul 07 15:44:37.468691 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:37.468596    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fvkrg" podUID="f0d07c9b-7fac-4665-b397-d996f6781c50"
Jul 07 15:44:37.468691 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:37.468601    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tx5v9"
Jul 07 15:44:37.468691 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:37.468682    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zbhgc" podUID="2a773f68-4a73-415d-abc9-3672aa14a254"
Jul 07 15:44:37.468834 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:37.468736    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tx5v9" podUID="e86522b9-3dc8-41c0-8d8a-40229b4795a7"
Jul 07 15:44:38.658676 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:38.658642    2566 generic.go:356] "Generic (PLEG): container finished" podID="0e3de181-9d7a-4f84-ae08-d018e710109c" containerID="677cf3c2337aaf3ae82f4484379980a422e42dbc09bc960482873ba10c411c1d" exitCode=0
Jul 07 15:44:38.659342 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:38.658684    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zbxwc" event={"ID":"0e3de181-9d7a-4f84-ae08-d018e710109c","Type":"ContainerDied","Data":"677cf3c2337aaf3ae82f4484379980a422e42dbc09bc960482873ba10c411c1d"}
Jul 07 15:44:39.468554 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:39.468520    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tx5v9"
Jul 07 15:44:39.468750 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:39.468639    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fvkrg"
Jul 07 15:44:39.468750 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:39.468655    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tx5v9" podUID="e86522b9-3dc8-41c0-8d8a-40229b4795a7"
Jul 07 15:44:39.468750 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:39.468666    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zbhgc"
Jul 07 15:44:39.468750 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:39.468725    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fvkrg" podUID="f0d07c9b-7fac-4665-b397-d996f6781c50"
Jul 07 15:44:39.468976 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:39.468793    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zbhgc" podUID="2a773f68-4a73-415d-abc9-3672aa14a254"
Jul 07 15:44:40.091783 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:40.091695    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a773f68-4a73-415d-abc9-3672aa14a254-metrics-certs\") pod \"network-metrics-daemon-zbhgc\" (UID: \"2a773f68-4a73-415d-abc9-3672aa14a254\") " pod="openshift-multus/network-metrics-daemon-zbhgc"
Jul 07 15:44:40.091783 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:40.091743    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f0d07c9b-7fac-4665-b397-d996f6781c50-original-pull-secret\") pod \"global-pull-secret-syncer-fvkrg\" (UID: \"f0d07c9b-7fac-4665-b397-d996f6781c50\") " pod="kube-system/global-pull-secret-syncer-fvkrg"
Jul 07 15:44:40.092222 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:40.091846    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 15:44:40.092222 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:40.091877    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 15:44:40.092222 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:40.091930    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a773f68-4a73-415d-abc9-3672aa14a254-metrics-certs podName:2a773f68-4a73-415d-abc9-3672aa14a254 nodeName:}" failed. No retries permitted until 2026-07-07 15:45:12.0919096 +0000 UTC m=+66.180335252 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2a773f68-4a73-415d-abc9-3672aa14a254-metrics-certs") pod "network-metrics-daemon-zbhgc" (UID: "2a773f68-4a73-415d-abc9-3672aa14a254") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 15:44:40.092222 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:40.091949    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0d07c9b-7fac-4665-b397-d996f6781c50-original-pull-secret podName:f0d07c9b-7fac-4665-b397-d996f6781c50 nodeName:}" failed. No retries permitted until 2026-07-07 15:45:12.091941968 +0000 UTC m=+66.180367616 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f0d07c9b-7fac-4665-b397-d996f6781c50-original-pull-secret") pod "global-pull-secret-syncer-fvkrg" (UID: "f0d07c9b-7fac-4665-b397-d996f6781c50") : object "kube-system"/"original-pull-secret" not registered
Jul 07 15:44:40.192928 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:40.192869    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4wkf\" (UniqueName: \"kubernetes.io/projected/e86522b9-3dc8-41c0-8d8a-40229b4795a7-kube-api-access-m4wkf\") pod \"network-check-target-tx5v9\" (UID: \"e86522b9-3dc8-41c0-8d8a-40229b4795a7\") " pod="openshift-network-diagnostics/network-check-target-tx5v9"
Jul 07 15:44:40.193113 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:40.193016    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 15:44:40.193113 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:40.193043    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 15:44:40.193113 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:40.193056    2566 projected.go:196] Error preparing data for projected volume kube-api-access-m4wkf for pod openshift-network-diagnostics/network-check-target-tx5v9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 15:44:40.193261 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:40.193115    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e86522b9-3dc8-41c0-8d8a-40229b4795a7-kube-api-access-m4wkf podName:e86522b9-3dc8-41c0-8d8a-40229b4795a7 nodeName:}" failed. No retries permitted until 2026-07-07 15:45:12.19309681 +0000 UTC m=+66.281522458 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-m4wkf" (UniqueName: "kubernetes.io/projected/e86522b9-3dc8-41c0-8d8a-40229b4795a7-kube-api-access-m4wkf") pod "network-check-target-tx5v9" (UID: "e86522b9-3dc8-41c0-8d8a-40229b4795a7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 15:44:41.469179 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:41.469142    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tx5v9"
Jul 07 15:44:41.469754 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:41.469262    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fvkrg"
Jul 07 15:44:41.469754 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:41.469266    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tx5v9" podUID="e86522b9-3dc8-41c0-8d8a-40229b4795a7"
Jul 07 15:44:41.469754 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:41.469365    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fvkrg" podUID="f0d07c9b-7fac-4665-b397-d996f6781c50"
Jul 07 15:44:41.469754 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:41.469267    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zbhgc"
Jul 07 15:44:41.469754 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:41.469450    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zbhgc" podUID="2a773f68-4a73-415d-abc9-3672aa14a254"
Jul 07 15:44:42.117478 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:42.117402    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-101.ec2.internal" event="NodeReady"
Jul 07 15:44:42.117624 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:42.117546    2566 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 07 15:44:42.187330 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:42.187272    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-t4pz9"]
Jul 07 15:44:42.191424 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:42.191393    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7c68df6494-4mrwk"]
Jul 07 15:44:42.191558 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:42.191514    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-t4pz9"
Jul 07 15:44:42.194094 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:42.194069    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 07 15:44:42.194217 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:42.194206    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 07 15:44:42.194442 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:42.194422    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-vwnd2\""
Jul 07 15:44:42.194442 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:42.194445    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 07 15:44:42.194712 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:42.194695    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-z47rg"]
Jul 07 15:44:42.194822 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:42.194808    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c68df6494-4mrwk"
Jul 07 15:44:42.197215 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:42.197087    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 07 15:44:42.197215 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:42.197142    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 07 15:44:42.197215 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:42.197160    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-fglhg\""
Jul 07 15:44:42.197405 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:42.197230    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 07 15:44:42.198263 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:42.198249    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-z47rg"
Jul 07 15:44:42.200672 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:42.200642    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-z47rg"]
Jul 07 15:44:42.200784 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:42.200769    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 07 15:44:42.200853 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:42.200796    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 07 15:44:42.200936 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:42.200865    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-j6trp\""
Jul 07 15:44:42.206857 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:42.206771    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-t4pz9"]
Jul 07 15:44:42.208762 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:42.207932    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7c68df6494-4mrwk"]
Jul 07 15:44:42.212152 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:42.212132    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 07 15:44:42.311298 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:42.311257    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5fc809cc-0c34-4431-be3d-1d08204edde6-metrics-tls\") pod \"dns-default-z47rg\" (UID: \"5fc809cc-0c34-4431-be3d-1d08204edde6\") " pod="openshift-dns/dns-default-z47rg"
Jul 07 15:44:42.311477 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:42.311357    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/54bc412f-9e52-45a2-b820-3dee0d9902d7-installation-pull-secrets\") pod \"image-registry-7c68df6494-4mrwk\" (UID: \"54bc412f-9e52-45a2-b820-3dee0d9902d7\") " pod="openshift-image-registry/image-registry-7c68df6494-4mrwk"
Jul 07 15:44:42.311477 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:42.311404    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/54bc412f-9e52-45a2-b820-3dee0d9902d7-bound-sa-token\") pod \"image-registry-7c68df6494-4mrwk\" (UID: \"54bc412f-9e52-45a2-b820-3dee0d9902d7\") " pod="openshift-image-registry/image-registry-7c68df6494-4mrwk"
Jul 07 15:44:42.311477 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:42.311454    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/54bc412f-9e52-45a2-b820-3dee0d9902d7-image-registry-private-configuration\") pod \"image-registry-7c68df6494-4mrwk\" (UID: \"54bc412f-9e52-45a2-b820-3dee0d9902d7\") " pod="openshift-image-registry/image-registry-7c68df6494-4mrwk"
Jul 07 15:44:42.311628 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:42.311506    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bl89k\" (UniqueName: \"kubernetes.io/projected/54bc412f-9e52-45a2-b820-3dee0d9902d7-kube-api-access-bl89k\") pod \"image-registry-7c68df6494-4mrwk\" (UID: \"54bc412f-9e52-45a2-b820-3dee0d9902d7\") " pod="openshift-image-registry/image-registry-7c68df6494-4mrwk"
Jul 07 15:44:42.311628 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:42.311555    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5fc809cc-0c34-4431-be3d-1d08204edde6-config-volume\") pod \"dns-default-z47rg\" (UID: \"5fc809cc-0c34-4431-be3d-1d08204edde6\") " pod="openshift-dns/dns-default-z47rg"
Jul 07 15:44:42.311628 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:42.311581    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/54bc412f-9e52-45a2-b820-3dee0d9902d7-registry-tls\") pod \"image-registry-7c68df6494-4mrwk\" (UID: \"54bc412f-9e52-45a2-b820-3dee0d9902d7\") " pod="openshift-image-registry/image-registry-7c68df6494-4mrwk"
Jul 07 15:44:42.311628 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:42.311604    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/54bc412f-9e52-45a2-b820-3dee0d9902d7-registry-certificates\") pod \"image-registry-7c68df6494-4mrwk\" (UID: \"54bc412f-9e52-45a2-b820-3dee0d9902d7\") " pod="openshift-image-registry/image-registry-7c68df6494-4mrwk"
Jul 07 15:44:42.311859 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:42.311629    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f12ca464-a7b0-4009-ba1f-dde12c17d1c3-cert\") pod \"ingress-canary-t4pz9\" (UID: \"f12ca464-a7b0-4009-ba1f-dde12c17d1c3\") " pod="openshift-ingress-canary/ingress-canary-t4pz9"
Jul 07 15:44:42.311859 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:42.311651    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtpwz\" (UniqueName: \"kubernetes.io/projected/f12ca464-a7b0-4009-ba1f-dde12c17d1c3-kube-api-access-vtpwz\") pod \"ingress-canary-t4pz9\" (UID: \"f12ca464-a7b0-4009-ba1f-dde12c17d1c3\") " pod="openshift-ingress-canary/ingress-canary-t4pz9"
Jul 07 15:44:42.311859 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:42.311684    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fc809cc-0c34-4431-be3d-1d08204edde6-tmp-dir\") pod \"dns-default-z47rg\" (UID: \"5fc809cc-0c34-4431-be3d-1d08204edde6\") " pod="openshift-dns/dns-default-z47rg"
Jul 07 15:44:42.311859 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:42.311723    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpx2d\" (UniqueName: \"kubernetes.io/projected/5fc809cc-0c34-4431-be3d-1d08204edde6-kube-api-access-cpx2d\") pod \"dns-default-z47rg\" (UID: \"5fc809cc-0c34-4431-be3d-1d08204edde6\") " pod="openshift-dns/dns-default-z47rg"
Jul 07 15:44:42.311859 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:42.311783    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/54bc412f-9e52-45a2-b820-3dee0d9902d7-ca-trust-extracted\") pod \"image-registry-7c68df6494-4mrwk\" (UID: \"54bc412f-9e52-45a2-b820-3dee0d9902d7\") " pod="openshift-image-registry/image-registry-7c68df6494-4mrwk"
Jul 07 15:44:42.311859 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:42.311828    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/54bc412f-9e52-45a2-b820-3dee0d9902d7-trusted-ca\") pod \"image-registry-7c68df6494-4mrwk\" (UID: \"54bc412f-9e52-45a2-b820-3dee0d9902d7\") " pod="openshift-image-registry/image-registry-7c68df6494-4mrwk"
Jul 07 15:44:42.412765 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:42.412682    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/54bc412f-9e52-45a2-b820-3dee0d9902d7-installation-pull-secrets\") pod \"image-registry-7c68df6494-4mrwk\" (UID: \"54bc412f-9e52-45a2-b820-3dee0d9902d7\") " pod="openshift-image-registry/image-registry-7c68df6494-4mrwk"
Jul 07 15:44:42.412765 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:42.412735    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/54bc412f-9e52-45a2-b820-3dee0d9902d7-bound-sa-token\") pod \"image-registry-7c68df6494-4mrwk\" (UID: \"54bc412f-9e52-45a2-b820-3dee0d9902d7\") " pod="openshift-image-registry/image-registry-7c68df6494-4mrwk"
Jul 07 15:44:42.413015 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:42.412783    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/54bc412f-9e52-45a2-b820-3dee0d9902d7-image-registry-private-configuration\") pod \"image-registry-7c68df6494-4mrwk\" (UID: \"54bc412f-9e52-45a2-b820-3dee0d9902d7\") " pod="openshift-image-registry/image-registry-7c68df6494-4mrwk"
Jul 07 15:44:42.413015 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:42.412807    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bl89k\" (UniqueName: \"kubernetes.io/projected/54bc412f-9e52-45a2-b820-3dee0d9902d7-kube-api-access-bl89k\") pod \"image-registry-7c68df6494-4mrwk\" (UID: \"54bc412f-9e52-45a2-b820-3dee0d9902d7\") " pod="openshift-image-registry/image-registry-7c68df6494-4mrwk"
Jul 07 15:44:42.413015 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:42.412841    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5fc809cc-0c34-4431-be3d-1d08204edde6-config-volume\") pod \"dns-default-z47rg\" (UID: \"5fc809cc-0c34-4431-be3d-1d08204edde6\") " pod="openshift-dns/dns-default-z47rg"
Jul 07 15:44:42.413015 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:42.412863    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/54bc412f-9e52-45a2-b820-3dee0d9902d7-registry-tls\") pod \"image-registry-7c68df6494-4mrwk\" (UID: \"54bc412f-9e52-45a2-b820-3dee0d9902d7\") " pod="openshift-image-registry/image-registry-7c68df6494-4mrwk"
Jul 07 15:44:42.413015 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:42.412897    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/54bc412f-9e52-45a2-b820-3dee0d9902d7-registry-certificates\") pod \"image-registry-7c68df6494-4mrwk\" (UID: \"54bc412f-9e52-45a2-b820-3dee0d9902d7\") " pod="openshift-image-registry/image-registry-7c68df6494-4mrwk"
Jul 07 15:44:42.413015 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:42.412926    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f12ca464-a7b0-4009-ba1f-dde12c17d1c3-cert\") pod \"ingress-canary-t4pz9\" (UID: \"f12ca464-a7b0-4009-ba1f-dde12c17d1c3\") " pod="openshift-ingress-canary/ingress-canary-t4pz9"
Jul 07 15:44:42.413015 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:42.412948    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtpwz\" (UniqueName: \"kubernetes.io/projected/f12ca464-a7b0-4009-ba1f-dde12c17d1c3-kube-api-access-vtpwz\") pod \"ingress-canary-t4pz9\" (UID: \"f12ca464-a7b0-4009-ba1f-dde12c17d1c3\") " pod="openshift-ingress-canary/ingress-canary-t4pz9"
Jul 07 15:44:42.413015 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:42.412975    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fc809cc-0c34-4431-be3d-1d08204edde6-tmp-dir\") pod \"dns-default-z47rg\" (UID: \"5fc809cc-0c34-4431-be3d-1d08204edde6\") " pod="openshift-dns/dns-default-z47rg"
Jul 07 15:44:42.413015 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:42.413019    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpx2d\" (UniqueName: \"kubernetes.io/projected/5fc809cc-0c34-4431-be3d-1d08204edde6-kube-api-access-cpx2d\") pod \"dns-default-z47rg\" (UID: \"5fc809cc-0c34-4431-be3d-1d08204edde6\") " pod="openshift-dns/dns-default-z47rg"
Jul 07 15:44:42.413387 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:42.413045    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/54bc412f-9e52-45a2-b820-3dee0d9902d7-ca-trust-extracted\") pod \"image-registry-7c68df6494-4mrwk\" (UID: \"54bc412f-9e52-45a2-b820-3dee0d9902d7\") " pod="openshift-image-registry/image-registry-7c68df6494-4mrwk"
Jul 07 15:44:42.413387 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:42.413058    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 07 15:44:42.413387 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:42.413071    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/54bc412f-9e52-45a2-b820-3dee0d9902d7-trusted-ca\") pod \"image-registry-7c68df6494-4mrwk\" (UID: \"54bc412f-9e52-45a2-b820-3dee0d9902d7\") " pod="openshift-image-registry/image-registry-7c68df6494-4mrwk"
Jul 07 15:44:42.413387 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:42.413080    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c68df6494-4mrwk: secret "image-registry-tls" not found
Jul 07 15:44:42.413387 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:42.413124    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5fc809cc-0c34-4431-be3d-1d08204edde6-metrics-tls\") pod \"dns-default-z47rg\" (UID: \"5fc809cc-0c34-4431-be3d-1d08204edde6\") " pod="openshift-dns/dns-default-z47rg"
Jul 07 15:44:42.413387 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:42.413157    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/54bc412f-9e52-45a2-b820-3dee0d9902d7-registry-tls podName:54bc412f-9e52-45a2-b820-3dee0d9902d7 nodeName:}" failed. No retries permitted until 2026-07-07 15:44:42.913136011 +0000 UTC m=+37.001561680 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/54bc412f-9e52-45a2-b820-3dee0d9902d7-registry-tls") pod "image-registry-7c68df6494-4mrwk" (UID: "54bc412f-9e52-45a2-b820-3dee0d9902d7") : secret "image-registry-tls" not found
Jul 07 15:44:42.413387 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:42.413217    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 15:44:42.413387 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:42.413272    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5fc809cc-0c34-4431-be3d-1d08204edde6-metrics-tls podName:5fc809cc-0c34-4431-be3d-1d08204edde6 nodeName:}" failed. No retries permitted until 2026-07-07 15:44:42.913257589 +0000 UTC m=+37.001683251 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5fc809cc-0c34-4431-be3d-1d08204edde6-metrics-tls") pod "dns-default-z47rg" (UID: "5fc809cc-0c34-4431-be3d-1d08204edde6") : secret "dns-default-metrics-tls" not found
Jul 07 15:44:42.413726 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:42.413409    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5fc809cc-0c34-4431-be3d-1d08204edde6-config-volume\") pod \"dns-default-z47rg\" (UID: \"5fc809cc-0c34-4431-be3d-1d08204edde6\") " pod="openshift-dns/dns-default-z47rg"
Jul 07 15:44:42.413726 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:42.413544    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fc809cc-0c34-4431-be3d-1d08204edde6-tmp-dir\") pod \"dns-default-z47rg\" (UID: \"5fc809cc-0c34-4431-be3d-1d08204edde6\") " pod="openshift-dns/dns-default-z47rg"
Jul 07 15:44:42.413726 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:42.413639    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/54bc412f-9e52-45a2-b820-3dee0d9902d7-ca-trust-extracted\") pod \"image-registry-7c68df6494-4mrwk\" (UID: \"54bc412f-9e52-45a2-b820-3dee0d9902d7\") " pod="openshift-image-registry/image-registry-7c68df6494-4mrwk"
Jul 07 15:44:42.414023 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:42.414000    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/54bc412f-9e52-45a2-b820-3dee0d9902d7-registry-certificates\") pod \"image-registry-7c68df6494-4mrwk\" (UID: \"54bc412f-9e52-45a2-b820-3dee0d9902d7\") " pod="openshift-image-registry/image-registry-7c68df6494-4mrwk"
Jul 07 15:44:42.414372 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:42.414345    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/54bc412f-9e52-45a2-b820-3dee0d9902d7-trusted-ca\") pod \"image-registry-7c68df6494-4mrwk\" (UID: \"54bc412f-9e52-45a2-b820-3dee0d9902d7\") " pod="openshift-image-registry/image-registry-7c68df6494-4mrwk"
Jul 07 15:44:42.417835 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:42.417814    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/54bc412f-9e52-45a2-b820-3dee0d9902d7-installation-pull-secrets\") pod \"image-registry-7c68df6494-4mrwk\" (UID: \"54bc412f-9e52-45a2-b820-3dee0d9902d7\") " pod="openshift-image-registry/image-registry-7c68df6494-4mrwk"
Jul 07 15:44:42.417947 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:42.417855    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f12ca464-a7b0-4009-ba1f-dde12c17d1c3-cert\") pod \"ingress-canary-t4pz9\" (UID: \"f12ca464-a7b0-4009-ba1f-dde12c17d1c3\") " pod="openshift-ingress-canary/ingress-canary-t4pz9"
Jul 07 15:44:42.417947 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:42.417875    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/54bc412f-9e52-45a2-b820-3dee0d9902d7-image-registry-private-configuration\") pod \"image-registry-7c68df6494-4mrwk\" (UID: \"54bc412f-9e52-45a2-b820-3dee0d9902d7\") " pod="openshift-image-registry/image-registry-7c68df6494-4mrwk"
Jul 07 15:44:42.424522 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:42.424381    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/54bc412f-9e52-45a2-b820-3dee0d9902d7-bound-sa-token\") pod \"image-registry-7c68df6494-4mrwk\" (UID: \"54bc412f-9e52-45a2-b820-3dee0d9902d7\") " pod="openshift-image-registry/image-registry-7c68df6494-4mrwk"
Jul 07 15:44:42.424645 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:42.424621    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bl89k\" (UniqueName: \"kubernetes.io/projected/54bc412f-9e52-45a2-b820-3dee0d9902d7-kube-api-access-bl89k\") pod \"image-registry-7c68df6494-4mrwk\" (UID: \"54bc412f-9e52-45a2-b820-3dee0d9902d7\") " pod="openshift-image-registry/image-registry-7c68df6494-4mrwk"
Jul 07 15:44:42.424859 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:42.424837    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpx2d\" (UniqueName: \"kubernetes.io/projected/5fc809cc-0c34-4431-be3d-1d08204edde6-kube-api-access-cpx2d\") pod \"dns-default-z47rg\" (UID: \"5fc809cc-0c34-4431-be3d-1d08204edde6\") " pod="openshift-dns/dns-default-z47rg"
Jul 07 15:44:42.425337 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:42.425292    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtpwz\" (UniqueName: \"kubernetes.io/projected/f12ca464-a7b0-4009-ba1f-dde12c17d1c3-kube-api-access-vtpwz\") pod \"ingress-canary-t4pz9\" (UID: \"f12ca464-a7b0-4009-ba1f-dde12c17d1c3\") " pod="openshift-ingress-canary/ingress-canary-t4pz9"
Jul 07 15:44:42.506701 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:42.506667    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-t4pz9"
Jul 07 15:44:42.916360 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:42.916325    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/54bc412f-9e52-45a2-b820-3dee0d9902d7-registry-tls\") pod \"image-registry-7c68df6494-4mrwk\" (UID: \"54bc412f-9e52-45a2-b820-3dee0d9902d7\") " pod="openshift-image-registry/image-registry-7c68df6494-4mrwk"
Jul 07 15:44:42.916549 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:42.916399    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5fc809cc-0c34-4431-be3d-1d08204edde6-metrics-tls\") pod \"dns-default-z47rg\" (UID: \"5fc809cc-0c34-4431-be3d-1d08204edde6\") " pod="openshift-dns/dns-default-z47rg"
Jul 07 15:44:42.916549 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:42.916487    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 07 15:44:42.916549 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:42.916504    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c68df6494-4mrwk: secret "image-registry-tls" not found
Jul 07 15:44:42.916549 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:42.916523    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 15:44:42.916735 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:42.916564    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/54bc412f-9e52-45a2-b820-3dee0d9902d7-registry-tls podName:54bc412f-9e52-45a2-b820-3dee0d9902d7 nodeName:}" failed. No retries permitted until 2026-07-07 15:44:43.916547962 +0000 UTC m=+38.004973624 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/54bc412f-9e52-45a2-b820-3dee0d9902d7-registry-tls") pod "image-registry-7c68df6494-4mrwk" (UID: "54bc412f-9e52-45a2-b820-3dee0d9902d7") : secret "image-registry-tls" not found
Jul 07 15:44:42.916735 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:42.916595    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5fc809cc-0c34-4431-be3d-1d08204edde6-metrics-tls podName:5fc809cc-0c34-4431-be3d-1d08204edde6 nodeName:}" failed. No retries permitted until 2026-07-07 15:44:43.916576475 +0000 UTC m=+38.005002123 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5fc809cc-0c34-4431-be3d-1d08204edde6-metrics-tls") pod "dns-default-z47rg" (UID: "5fc809cc-0c34-4431-be3d-1d08204edde6") : secret "dns-default-metrics-tls" not found
Jul 07 15:44:43.435692 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:43.435517    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-t4pz9"]
Jul 07 15:44:43.439808 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:43.439782    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf12ca464_a7b0_4009_ba1f_dde12c17d1c3.slice/crio-1aab62d010ec3a6a1748e8641a2c0961c26dc79b580e9f68f583792e375b5dd2 WatchSource:0}: Error finding container 1aab62d010ec3a6a1748e8641a2c0961c26dc79b580e9f68f583792e375b5dd2: Status 404 returned error can't find the container with id 1aab62d010ec3a6a1748e8641a2c0961c26dc79b580e9f68f583792e375b5dd2
Jul 07 15:44:43.468346 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:43.468321    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zbhgc"
Jul 07 15:44:43.468452 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:43.468321    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tx5v9"
Jul 07 15:44:43.468521 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:43.468325    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fvkrg"
Jul 07 15:44:43.471023 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:43.471003    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 07 15:44:43.471114 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:43.471005    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 07 15:44:43.471756 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:43.471735    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-wcvdn\""
Jul 07 15:44:43.471853 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:43.471818    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ctzps\""
Jul 07 15:44:43.471924 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:43.471880    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 07 15:44:43.472001 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:43.471944    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 07 15:44:43.672830 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:43.672795    2566 generic.go:356] "Generic (PLEG): container finished" podID="0e3de181-9d7a-4f84-ae08-d018e710109c" containerID="1975d74de433a0396a96ea2b485f6a25fa0cb37916fd0569e3490802bf8a5ab3" exitCode=0
Jul 07 15:44:43.673290 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:43.672881    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zbxwc" event={"ID":"0e3de181-9d7a-4f84-ae08-d018e710109c","Type":"ContainerDied","Data":"1975d74de433a0396a96ea2b485f6a25fa0cb37916fd0569e3490802bf8a5ab3"}
Jul 07 15:44:43.673971 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:43.673949    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-t4pz9" event={"ID":"f12ca464-a7b0-4009-ba1f-dde12c17d1c3","Type":"ContainerStarted","Data":"1aab62d010ec3a6a1748e8641a2c0961c26dc79b580e9f68f583792e375b5dd2"}
Jul 07 15:44:43.923777 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:43.923698    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/54bc412f-9e52-45a2-b820-3dee0d9902d7-registry-tls\") pod \"image-registry-7c68df6494-4mrwk\" (UID: \"54bc412f-9e52-45a2-b820-3dee0d9902d7\") " pod="openshift-image-registry/image-registry-7c68df6494-4mrwk"
Jul 07 15:44:43.923777 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:43.923766    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5fc809cc-0c34-4431-be3d-1d08204edde6-metrics-tls\") pod \"dns-default-z47rg\" (UID: \"5fc809cc-0c34-4431-be3d-1d08204edde6\") " pod="openshift-dns/dns-default-z47rg"
Jul 07 15:44:43.923968 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:43.923845    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 07 15:44:43.923968 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:43.923867    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c68df6494-4mrwk: secret "image-registry-tls" not found
Jul 07 15:44:43.923968 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:43.923852    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 15:44:43.923968 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:43.923937    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/54bc412f-9e52-45a2-b820-3dee0d9902d7-registry-tls podName:54bc412f-9e52-45a2-b820-3dee0d9902d7 nodeName:}" failed. No retries permitted until 2026-07-07 15:44:45.923920617 +0000 UTC m=+40.012346271 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/54bc412f-9e52-45a2-b820-3dee0d9902d7-registry-tls") pod "image-registry-7c68df6494-4mrwk" (UID: "54bc412f-9e52-45a2-b820-3dee0d9902d7") : secret "image-registry-tls" not found
Jul 07 15:44:43.923968 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:43.923954    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5fc809cc-0c34-4431-be3d-1d08204edde6-metrics-tls podName:5fc809cc-0c34-4431-be3d-1d08204edde6 nodeName:}" failed. No retries permitted until 2026-07-07 15:44:45.923948535 +0000 UTC m=+40.012374182 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5fc809cc-0c34-4431-be3d-1d08204edde6-metrics-tls") pod "dns-default-z47rg" (UID: "5fc809cc-0c34-4431-be3d-1d08204edde6") : secret "dns-default-metrics-tls" not found
Jul 07 15:44:44.680156 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:44.680120    2566 generic.go:356] "Generic (PLEG): container finished" podID="0e3de181-9d7a-4f84-ae08-d018e710109c" containerID="d5807dacd9baaa81215688c21b2fcac9133b22b1d4a5adf2e098f54b5b34e094" exitCode=0
Jul 07 15:44:44.680536 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:44.680186    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zbxwc" event={"ID":"0e3de181-9d7a-4f84-ae08-d018e710109c","Type":"ContainerDied","Data":"d5807dacd9baaa81215688c21b2fcac9133b22b1d4a5adf2e098f54b5b34e094"}
Jul 07 15:44:45.687110 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:45.686831    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zbxwc" event={"ID":"0e3de181-9d7a-4f84-ae08-d018e710109c","Type":"ContainerStarted","Data":"7126d5fdd181de97921a8651d5be8ec6fff0d5d27c6abe87ca03ae1fb5957363"}
Jul 07 15:44:45.716639 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:45.716582    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-zbxwc" podStartSLOduration=15.717711387 podStartE2EDuration="38.71656442s" totalImagesPullingTime="22.998853033s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 15:44:07 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:44:45.715363798 +0000 UTC m=+39.803789470" watchObservedRunningTime="2026-07-07 15:44:45.71656442 +0000 UTC m=+39.804990089"
Jul 07 15:44:45.941721 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:45.941627    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/54bc412f-9e52-45a2-b820-3dee0d9902d7-registry-tls\") pod \"image-registry-7c68df6494-4mrwk\" (UID: \"54bc412f-9e52-45a2-b820-3dee0d9902d7\") " pod="openshift-image-registry/image-registry-7c68df6494-4mrwk"
Jul 07 15:44:45.941873 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:45.941731    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5fc809cc-0c34-4431-be3d-1d08204edde6-metrics-tls\") pod \"dns-default-z47rg\" (UID: \"5fc809cc-0c34-4431-be3d-1d08204edde6\") " pod="openshift-dns/dns-default-z47rg"
Jul 07 15:44:45.941873 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:45.941796    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 07 15:44:45.941873 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:45.941813    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c68df6494-4mrwk: secret "image-registry-tls" not found
Jul 07 15:44:45.941873 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:45.941829    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 15:44:45.941873 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:45.941874    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/54bc412f-9e52-45a2-b820-3dee0d9902d7-registry-tls podName:54bc412f-9e52-45a2-b820-3dee0d9902d7 nodeName:}" failed. No retries permitted until 2026-07-07 15:44:49.941859221 +0000 UTC m=+44.030284876 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/54bc412f-9e52-45a2-b820-3dee0d9902d7-registry-tls") pod "image-registry-7c68df6494-4mrwk" (UID: "54bc412f-9e52-45a2-b820-3dee0d9902d7") : secret "image-registry-tls" not found
Jul 07 15:44:45.942124 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:45.941902    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5fc809cc-0c34-4431-be3d-1d08204edde6-metrics-tls podName:5fc809cc-0c34-4431-be3d-1d08204edde6 nodeName:}" failed. No retries permitted until 2026-07-07 15:44:49.94189593 +0000 UTC m=+44.030321579 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5fc809cc-0c34-4431-be3d-1d08204edde6-metrics-tls") pod "dns-default-z47rg" (UID: "5fc809cc-0c34-4431-be3d-1d08204edde6") : secret "dns-default-metrics-tls" not found
Jul 07 15:44:46.690746 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:46.690651    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-t4pz9" event={"ID":"f12ca464-a7b0-4009-ba1f-dde12c17d1c3","Type":"ContainerStarted","Data":"6ebbefb71d144c1c3207eefec07531b3ea6579ab9599697b7957c282b67577e5"}
Jul 07 15:44:46.710738 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:46.710688    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-t4pz9" podStartSLOduration=1.7670011479999999 podStartE2EDuration="4.710673612s" totalImagesPullingTime="2.943672464s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:44:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:44:46.709520142 +0000 UTC m=+40.797945812" watchObservedRunningTime="2026-07-07 15:44:46.710673612 +0000 UTC m=+40.799099283"
Jul 07 15:44:47.348611 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:47.348582    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-58tdv"]
Jul 07 15:44:47.370755 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:47.370725    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-58tdv"]
Jul 07 15:44:47.370919 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:47.370832    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-58tdv"
Jul 07 15:44:47.374664 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:47.374641    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-7zqlw\""
Jul 07 15:44:47.374922 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:47.374905    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 07 15:44:47.375418 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:47.375404    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 07 15:44:47.415698 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:47.415670    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-bggdc"]
Jul 07 15:44:47.436837 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:47.436808    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bggdc"
Jul 07 15:44:47.440364 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:47.440342    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 07 15:44:47.440479 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:47.440408    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qsn42\""
Jul 07 15:44:47.440963 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:47.440947    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 07 15:44:47.441020 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:47.440974    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 07 15:44:47.441159 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:47.441146    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 07 15:44:47.448954 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:47.448931    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bggdc"]
Jul 07 15:44:47.552505 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:47.552470    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhg48\" (UniqueName: \"kubernetes.io/projected/8a9671f3-44c2-473a-a8df-fd41bd2f4235-kube-api-access-xhg48\") pod \"migrator-d57d58cc9-58tdv\" (UID: \"8a9671f3-44c2-473a-a8df-fd41bd2f4235\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-58tdv"
Jul 07 15:44:47.552691 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:47.552511    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b7a06c58-5567-40aa-b24e-c5f71943f570-crio-socket\") pod \"insights-runtime-extractor-bggdc\" (UID: \"b7a06c58-5567-40aa-b24e-c5f71943f570\") " pod="openshift-insights/insights-runtime-extractor-bggdc"
Jul 07 15:44:47.552691 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:47.552532    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b7a06c58-5567-40aa-b24e-c5f71943f570-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bggdc\" (UID: \"b7a06c58-5567-40aa-b24e-c5f71943f570\") " pod="openshift-insights/insights-runtime-extractor-bggdc"
Jul 07 15:44:47.552691 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:47.552601    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b7a06c58-5567-40aa-b24e-c5f71943f570-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bggdc\" (UID: \"b7a06c58-5567-40aa-b24e-c5f71943f570\") " pod="openshift-insights/insights-runtime-extractor-bggdc"
Jul 07 15:44:47.552691 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:47.552628    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9mnh\" (UniqueName: \"kubernetes.io/projected/b7a06c58-5567-40aa-b24e-c5f71943f570-kube-api-access-l9mnh\") pod \"insights-runtime-extractor-bggdc\" (UID: \"b7a06c58-5567-40aa-b24e-c5f71943f570\") " pod="openshift-insights/insights-runtime-extractor-bggdc"
Jul 07 15:44:47.552908 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:47.552775    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b7a06c58-5567-40aa-b24e-c5f71943f570-data-volume\") pod \"insights-runtime-extractor-bggdc\" (UID: \"b7a06c58-5567-40aa-b24e-c5f71943f570\") " pod="openshift-insights/insights-runtime-extractor-bggdc"
Jul 07 15:44:47.654146 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:47.654058    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b7a06c58-5567-40aa-b24e-c5f71943f570-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bggdc\" (UID: \"b7a06c58-5567-40aa-b24e-c5f71943f570\") " pod="openshift-insights/insights-runtime-extractor-bggdc"
Jul 07 15:44:47.654146 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:47.654097    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9mnh\" (UniqueName: \"kubernetes.io/projected/b7a06c58-5567-40aa-b24e-c5f71943f570-kube-api-access-l9mnh\") pod \"insights-runtime-extractor-bggdc\" (UID: \"b7a06c58-5567-40aa-b24e-c5f71943f570\") " pod="openshift-insights/insights-runtime-extractor-bggdc"
Jul 07 15:44:47.654358 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:47.654218    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 07 15:44:47.654358 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:47.654257    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b7a06c58-5567-40aa-b24e-c5f71943f570-data-volume\") pod \"insights-runtime-extractor-bggdc\" (UID: \"b7a06c58-5567-40aa-b24e-c5f71943f570\") " pod="openshift-insights/insights-runtime-extractor-bggdc"
Jul 07 15:44:47.654358 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:47.654291    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b7a06c58-5567-40aa-b24e-c5f71943f570-kube-rbac-proxy-cm podName:b7a06c58-5567-40aa-b24e-c5f71943f570 nodeName:}" failed. No retries permitted until 2026-07-07 15:44:48.154272664 +0000 UTC m=+42.242698326 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b7a06c58-5567-40aa-b24e-c5f71943f570-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bggdc" (UID: "b7a06c58-5567-40aa-b24e-c5f71943f570") : configmap "kube-rbac-proxy" not found
Jul 07 15:44:47.654512 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:47.654364    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhg48\" (UniqueName: \"kubernetes.io/projected/8a9671f3-44c2-473a-a8df-fd41bd2f4235-kube-api-access-xhg48\") pod \"migrator-d57d58cc9-58tdv\" (UID: \"8a9671f3-44c2-473a-a8df-fd41bd2f4235\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-58tdv"
Jul 07 15:44:47.654512 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:47.654416    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b7a06c58-5567-40aa-b24e-c5f71943f570-crio-socket\") pod \"insights-runtime-extractor-bggdc\" (UID: \"b7a06c58-5567-40aa-b24e-c5f71943f570\") " pod="openshift-insights/insights-runtime-extractor-bggdc"
Jul 07 15:44:47.654512 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:47.654445    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b7a06c58-5567-40aa-b24e-c5f71943f570-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bggdc\" (UID: \"b7a06c58-5567-40aa-b24e-c5f71943f570\") " pod="openshift-insights/insights-runtime-extractor-bggdc"
Jul 07 15:44:47.654639 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:47.654552    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 07 15:44:47.654639 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:47.654565    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b7a06c58-5567-40aa-b24e-c5f71943f570-data-volume\") pod \"insights-runtime-extractor-bggdc\" (UID: \"b7a06c58-5567-40aa-b24e-c5f71943f570\") " pod="openshift-insights/insights-runtime-extractor-bggdc"
Jul 07 15:44:47.654639 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:47.654577    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b7a06c58-5567-40aa-b24e-c5f71943f570-crio-socket\") pod \"insights-runtime-extractor-bggdc\" (UID: \"b7a06c58-5567-40aa-b24e-c5f71943f570\") " pod="openshift-insights/insights-runtime-extractor-bggdc"
Jul 07 15:44:47.654639 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:47.654593    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7a06c58-5567-40aa-b24e-c5f71943f570-insights-runtime-extractor-tls podName:b7a06c58-5567-40aa-b24e-c5f71943f570 nodeName:}" failed. No retries permitted until 2026-07-07 15:44:48.154579319 +0000 UTC m=+42.243004970 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b7a06c58-5567-40aa-b24e-c5f71943f570-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bggdc" (UID: "b7a06c58-5567-40aa-b24e-c5f71943f570") : secret "insights-runtime-extractor-tls" not found
Jul 07 15:44:47.664057 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:47.664033    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9mnh\" (UniqueName: \"kubernetes.io/projected/b7a06c58-5567-40aa-b24e-c5f71943f570-kube-api-access-l9mnh\") pod \"insights-runtime-extractor-bggdc\" (UID: \"b7a06c58-5567-40aa-b24e-c5f71943f570\") " pod="openshift-insights/insights-runtime-extractor-bggdc"
Jul 07 15:44:47.664170 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:47.664076    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhg48\" (UniqueName: \"kubernetes.io/projected/8a9671f3-44c2-473a-a8df-fd41bd2f4235-kube-api-access-xhg48\") pod \"migrator-d57d58cc9-58tdv\" (UID: \"8a9671f3-44c2-473a-a8df-fd41bd2f4235\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-58tdv"
Jul 07 15:44:47.679040 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:47.679018    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-58tdv"
Jul 07 15:44:47.816064 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:47.816026    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-58tdv"]
Jul 07 15:44:47.818817 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:47.818781    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a9671f3_44c2_473a_a8df_fd41bd2f4235.slice/crio-a6b53cee064991332d0845e8f0f07a02482d1705b6a2431b3a3ecbbb906cc7da WatchSource:0}: Error finding container a6b53cee064991332d0845e8f0f07a02482d1705b6a2431b3a3ecbbb906cc7da: Status 404 returned error can't find the container with id a6b53cee064991332d0845e8f0f07a02482d1705b6a2431b3a3ecbbb906cc7da
Jul 07 15:44:48.158961 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:48.158917    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b7a06c58-5567-40aa-b24e-c5f71943f570-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bggdc\" (UID: \"b7a06c58-5567-40aa-b24e-c5f71943f570\") " pod="openshift-insights/insights-runtime-extractor-bggdc"
Jul 07 15:44:48.159162 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:48.158975    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b7a06c58-5567-40aa-b24e-c5f71943f570-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bggdc\" (UID: \"b7a06c58-5567-40aa-b24e-c5f71943f570\") " pod="openshift-insights/insights-runtime-extractor-bggdc"
Jul 07 15:44:48.159162 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:48.159069    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 07 15:44:48.159162 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:48.159102    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 07 15:44:48.159162 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:48.159160    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7a06c58-5567-40aa-b24e-c5f71943f570-insights-runtime-extractor-tls podName:b7a06c58-5567-40aa-b24e-c5f71943f570 nodeName:}" failed. No retries permitted until 2026-07-07 15:44:49.159140663 +0000 UTC m=+43.247566314 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b7a06c58-5567-40aa-b24e-c5f71943f570-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bggdc" (UID: "b7a06c58-5567-40aa-b24e-c5f71943f570") : secret "insights-runtime-extractor-tls" not found
Jul 07 15:44:48.159381 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:48.159181    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b7a06c58-5567-40aa-b24e-c5f71943f570-kube-rbac-proxy-cm podName:b7a06c58-5567-40aa-b24e-c5f71943f570 nodeName:}" failed. No retries permitted until 2026-07-07 15:44:49.159171409 +0000 UTC m=+43.247597060 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b7a06c58-5567-40aa-b24e-c5f71943f570-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bggdc" (UID: "b7a06c58-5567-40aa-b24e-c5f71943f570") : configmap "kube-rbac-proxy" not found
Jul 07 15:44:48.696554 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:48.696521    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-58tdv" event={"ID":"8a9671f3-44c2-473a-a8df-fd41bd2f4235","Type":"ContainerStarted","Data":"a6b53cee064991332d0845e8f0f07a02482d1705b6a2431b3a3ecbbb906cc7da"}
Jul 07 15:44:49.167124 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:49.167085    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b7a06c58-5567-40aa-b24e-c5f71943f570-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bggdc\" (UID: \"b7a06c58-5567-40aa-b24e-c5f71943f570\") " pod="openshift-insights/insights-runtime-extractor-bggdc"
Jul 07 15:44:49.167502 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:49.167219    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 07 15:44:49.167502 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:49.167256    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b7a06c58-5567-40aa-b24e-c5f71943f570-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bggdc\" (UID: \"b7a06c58-5567-40aa-b24e-c5f71943f570\") " pod="openshift-insights/insights-runtime-extractor-bggdc"
Jul 07 15:44:49.167502 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:49.167281    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b7a06c58-5567-40aa-b24e-c5f71943f570-kube-rbac-proxy-cm podName:b7a06c58-5567-40aa-b24e-c5f71943f570 nodeName:}" failed. No retries permitted until 2026-07-07 15:44:51.167265451 +0000 UTC m=+45.255691099 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b7a06c58-5567-40aa-b24e-c5f71943f570-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bggdc" (UID: "b7a06c58-5567-40aa-b24e-c5f71943f570") : configmap "kube-rbac-proxy" not found
Jul 07 15:44:49.167502 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:49.167359    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 07 15:44:49.167502 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:49.167402    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7a06c58-5567-40aa-b24e-c5f71943f570-insights-runtime-extractor-tls podName:b7a06c58-5567-40aa-b24e-c5f71943f570 nodeName:}" failed. No retries permitted until 2026-07-07 15:44:51.167391207 +0000 UTC m=+45.255816855 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b7a06c58-5567-40aa-b24e-c5f71943f570-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bggdc" (UID: "b7a06c58-5567-40aa-b24e-c5f71943f570") : secret "insights-runtime-extractor-tls" not found
Jul 07 15:44:49.591700 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:49.591676    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6cgnn_22f1cba0-dd6e-4f5a-8d37-d83d1f938ace/dns-node-resolver/0.log"
Jul 07 15:44:49.700501 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:49.700474    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-58tdv" event={"ID":"8a9671f3-44c2-473a-a8df-fd41bd2f4235","Type":"ContainerStarted","Data":"cb5f86e5144ecd65583fbbff3cec01dde4c4b30789cddc885f22c2732cbf7123"}
Jul 07 15:44:49.700646 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:49.700574    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-58tdv" event={"ID":"8a9671f3-44c2-473a-a8df-fd41bd2f4235","Type":"ContainerStarted","Data":"d376d311a9e58cd3a514f70a11e5786736e0a01de652d0de8f3a8524d4d7b3d7"}
Jul 07 15:44:49.717560 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:49.717508    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-58tdv" podStartSLOduration=1.120255188 podStartE2EDuration="2.717491638s" totalImagesPullingTime="1.59723645s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:44:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:44:49.716812278 +0000 UTC m=+43.805237949" watchObservedRunningTime="2026-07-07 15:44:49.717491638 +0000 UTC m=+43.805917307"
Jul 07 15:44:49.974201 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:49.974121    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5fc809cc-0c34-4431-be3d-1d08204edde6-metrics-tls\") pod \"dns-default-z47rg\" (UID: \"5fc809cc-0c34-4431-be3d-1d08204edde6\") " pod="openshift-dns/dns-default-z47rg"
Jul 07 15:44:49.974201 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:49.974196    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/54bc412f-9e52-45a2-b820-3dee0d9902d7-registry-tls\") pod \"image-registry-7c68df6494-4mrwk\" (UID: \"54bc412f-9e52-45a2-b820-3dee0d9902d7\") " pod="openshift-image-registry/image-registry-7c68df6494-4mrwk"
Jul 07 15:44:49.974395 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:49.974276    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 15:44:49.974395 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:49.974329    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5fc809cc-0c34-4431-be3d-1d08204edde6-metrics-tls podName:5fc809cc-0c34-4431-be3d-1d08204edde6 nodeName:}" failed. No retries permitted until 2026-07-07 15:44:57.974315058 +0000 UTC m=+52.062740707 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5fc809cc-0c34-4431-be3d-1d08204edde6-metrics-tls") pod "dns-default-z47rg" (UID: "5fc809cc-0c34-4431-be3d-1d08204edde6") : secret "dns-default-metrics-tls" not found
Jul 07 15:44:49.974395 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:49.974375    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 07 15:44:49.974395 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:49.974396    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c68df6494-4mrwk: secret "image-registry-tls" not found
Jul 07 15:44:49.974561 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:49.974456    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/54bc412f-9e52-45a2-b820-3dee0d9902d7-registry-tls podName:54bc412f-9e52-45a2-b820-3dee0d9902d7 nodeName:}" failed. No retries permitted until 2026-07-07 15:44:57.97443789 +0000 UTC m=+52.062863554 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/54bc412f-9e52-45a2-b820-3dee0d9902d7-registry-tls") pod "image-registry-7c68df6494-4mrwk" (UID: "54bc412f-9e52-45a2-b820-3dee0d9902d7") : secret "image-registry-tls" not found
Jul 07 15:44:50.790838 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:50.790811    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-shpjq_ff08ea94-65cb-4ca1-a272-3de7627d10b8/node-ca/0.log"
Jul 07 15:44:51.185448 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:51.185353    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b7a06c58-5567-40aa-b24e-c5f71943f570-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bggdc\" (UID: \"b7a06c58-5567-40aa-b24e-c5f71943f570\") " pod="openshift-insights/insights-runtime-extractor-bggdc"
Jul 07 15:44:51.185448 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:51.185404    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b7a06c58-5567-40aa-b24e-c5f71943f570-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bggdc\" (UID: \"b7a06c58-5567-40aa-b24e-c5f71943f570\") " pod="openshift-insights/insights-runtime-extractor-bggdc"
Jul 07 15:44:51.185671 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:51.185468    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 07 15:44:51.185671 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:51.185519    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 07 15:44:51.185671 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:51.185584    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7a06c58-5567-40aa-b24e-c5f71943f570-insights-runtime-extractor-tls podName:b7a06c58-5567-40aa-b24e-c5f71943f570 nodeName:}" failed. No retries permitted until 2026-07-07 15:44:55.185564487 +0000 UTC m=+49.273990141 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b7a06c58-5567-40aa-b24e-c5f71943f570-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bggdc" (UID: "b7a06c58-5567-40aa-b24e-c5f71943f570") : secret "insights-runtime-extractor-tls" not found
Jul 07 15:44:51.185671 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:51.185601    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b7a06c58-5567-40aa-b24e-c5f71943f570-kube-rbac-proxy-cm podName:b7a06c58-5567-40aa-b24e-c5f71943f570 nodeName:}" failed. No retries permitted until 2026-07-07 15:44:55.185592847 +0000 UTC m=+49.274018494 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b7a06c58-5567-40aa-b24e-c5f71943f570-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bggdc" (UID: "b7a06c58-5567-40aa-b24e-c5f71943f570") : configmap "kube-rbac-proxy" not found
Jul 07 15:44:51.590836 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:51.590808    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-t4pz9_f12ca464-a7b0-4009-ba1f-dde12c17d1c3/serve-healthcheck-canary/0.log"
Jul 07 15:44:51.791564 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:51.791531    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-58tdv_8a9671f3-44c2-473a-a8df-fd41bd2f4235/migrator/0.log"
Jul 07 15:44:51.990228 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:51.990197    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-58tdv_8a9671f3-44c2-473a-a8df-fd41bd2f4235/graceful-termination/0.log"
Jul 07 15:44:55.220852 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:55.220822    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b7a06c58-5567-40aa-b24e-c5f71943f570-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bggdc\" (UID: \"b7a06c58-5567-40aa-b24e-c5f71943f570\") " pod="openshift-insights/insights-runtime-extractor-bggdc"
Jul 07 15:44:55.221262 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:55.220954    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b7a06c58-5567-40aa-b24e-c5f71943f570-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bggdc\" (UID: \"b7a06c58-5567-40aa-b24e-c5f71943f570\") " pod="openshift-insights/insights-runtime-extractor-bggdc"
Jul 07 15:44:55.221262 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:55.220968    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 07 15:44:55.221262 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:55.221032    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b7a06c58-5567-40aa-b24e-c5f71943f570-kube-rbac-proxy-cm podName:b7a06c58-5567-40aa-b24e-c5f71943f570 nodeName:}" failed. No retries permitted until 2026-07-07 15:45:03.221014943 +0000 UTC m=+57.309440595 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b7a06c58-5567-40aa-b24e-c5f71943f570-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bggdc" (UID: "b7a06c58-5567-40aa-b24e-c5f71943f570") : configmap "kube-rbac-proxy" not found
Jul 07 15:44:55.221262 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:55.221053    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 07 15:44:55.221262 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:44:55.221087    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7a06c58-5567-40aa-b24e-c5f71943f570-insights-runtime-extractor-tls podName:b7a06c58-5567-40aa-b24e-c5f71943f570 nodeName:}" failed. No retries permitted until 2026-07-07 15:45:03.221076718 +0000 UTC m=+57.309502370 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b7a06c58-5567-40aa-b24e-c5f71943f570-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bggdc" (UID: "b7a06c58-5567-40aa-b24e-c5f71943f570") : secret "insights-runtime-extractor-tls" not found
Jul 07 15:44:58.044632 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:58.044591    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/54bc412f-9e52-45a2-b820-3dee0d9902d7-registry-tls\") pod \"image-registry-7c68df6494-4mrwk\" (UID: \"54bc412f-9e52-45a2-b820-3dee0d9902d7\") " pod="openshift-image-registry/image-registry-7c68df6494-4mrwk"
Jul 07 15:44:58.045092 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:58.044820    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5fc809cc-0c34-4431-be3d-1d08204edde6-metrics-tls\") pod \"dns-default-z47rg\" (UID: \"5fc809cc-0c34-4431-be3d-1d08204edde6\") " pod="openshift-dns/dns-default-z47rg"
Jul 07 15:44:58.047112 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:58.047082    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/54bc412f-9e52-45a2-b820-3dee0d9902d7-registry-tls\") pod \"image-registry-7c68df6494-4mrwk\" (UID: \"54bc412f-9e52-45a2-b820-3dee0d9902d7\") " pod="openshift-image-registry/image-registry-7c68df6494-4mrwk"
Jul 07 15:44:58.047212 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:58.047089    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5fc809cc-0c34-4431-be3d-1d08204edde6-metrics-tls\") pod \"dns-default-z47rg\" (UID: \"5fc809cc-0c34-4431-be3d-1d08204edde6\") " pod="openshift-dns/dns-default-z47rg"
Jul 07 15:44:58.116215 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:58.116176    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c68df6494-4mrwk"
Jul 07 15:44:58.121334 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:58.121310    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-z47rg"
Jul 07 15:44:58.263565 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:58.263535    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7c68df6494-4mrwk"]
Jul 07 15:44:58.266677 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:58.266650    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54bc412f_9e52_45a2_b820_3dee0d9902d7.slice/crio-1e9a99dac28b13beb4efd850bc960621fc758b5db40076b0b9ac2d7c97326b9e WatchSource:0}: Error finding container 1e9a99dac28b13beb4efd850bc960621fc758b5db40076b0b9ac2d7c97326b9e: Status 404 returned error can't find the container with id 1e9a99dac28b13beb4efd850bc960621fc758b5db40076b0b9ac2d7c97326b9e
Jul 07 15:44:58.281479 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:58.281453    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-z47rg"]
Jul 07 15:44:58.284526 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:44:58.284500    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fc809cc_0c34_4431_be3d_1d08204edde6.slice/crio-7862eea7fc4cc1ff6895697e2aff257e51b7e33e3eddb855e893eecef69d36d6 WatchSource:0}: Error finding container 7862eea7fc4cc1ff6895697e2aff257e51b7e33e3eddb855e893eecef69d36d6: Status 404 returned error can't find the container with id 7862eea7fc4cc1ff6895697e2aff257e51b7e33e3eddb855e893eecef69d36d6
Jul 07 15:44:58.721178 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:58.721139    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-z47rg" event={"ID":"5fc809cc-0c34-4431-be3d-1d08204edde6","Type":"ContainerStarted","Data":"7862eea7fc4cc1ff6895697e2aff257e51b7e33e3eddb855e893eecef69d36d6"}
Jul 07 15:44:58.722743 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:58.722715    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c68df6494-4mrwk" event={"ID":"54bc412f-9e52-45a2-b820-3dee0d9902d7","Type":"ContainerStarted","Data":"f2f62de2796a7d39c708647630c9c3465ea65d3682c5a4be91a347a7f4046135"}
Jul 07 15:44:58.722865 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:58.722763    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c68df6494-4mrwk" event={"ID":"54bc412f-9e52-45a2-b820-3dee0d9902d7","Type":"ContainerStarted","Data":"1e9a99dac28b13beb4efd850bc960621fc758b5db40076b0b9ac2d7c97326b9e"}
Jul 07 15:44:58.722865 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:58.722804    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7c68df6494-4mrwk"
Jul 07 15:44:58.743512 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:44:58.743451    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7c68df6494-4mrwk" podStartSLOduration=18.743433203 podStartE2EDuration="18.743433203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:44:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:44:58.743362238 +0000 UTC m=+52.831787908" watchObservedRunningTime="2026-07-07 15:44:58.743433203 +0000 UTC m=+52.831858872"
Jul 07 15:45:01.731915 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:01.731868    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-z47rg" event={"ID":"5fc809cc-0c34-4431-be3d-1d08204edde6","Type":"ContainerStarted","Data":"1db1e87a9f9a10959158edd27f43475068a87915c83659a70c8ab346b49fca9d"}
Jul 07 15:45:01.732308 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:01.731936    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-z47rg" event={"ID":"5fc809cc-0c34-4431-be3d-1d08204edde6","Type":"ContainerStarted","Data":"f5081c14b6cee300a3a653c8ea89ff4543d338b97804ecfeb869e1eb8792ce09"}
Jul 07 15:45:01.732308 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:01.732033    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-z47rg"
Jul 07 15:45:01.751273 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:01.751229    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-z47rg" podStartSLOduration=17.402547773 podStartE2EDuration="19.75121475s" totalImagesPullingTime="2.348666977s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:44:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:45:01.75007938 +0000 UTC m=+55.838505051" watchObservedRunningTime="2026-07-07 15:45:01.75121475 +0000 UTC m=+55.839640420"
Jul 07 15:45:03.286512 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:03.286470    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b7a06c58-5567-40aa-b24e-c5f71943f570-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bggdc\" (UID: \"b7a06c58-5567-40aa-b24e-c5f71943f570\") " pod="openshift-insights/insights-runtime-extractor-bggdc"
Jul 07 15:45:03.286512 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:03.286526    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b7a06c58-5567-40aa-b24e-c5f71943f570-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bggdc\" (UID: \"b7a06c58-5567-40aa-b24e-c5f71943f570\") " pod="openshift-insights/insights-runtime-extractor-bggdc"
Jul 07 15:45:03.287156 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:03.287027    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b7a06c58-5567-40aa-b24e-c5f71943f570-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bggdc\" (UID: \"b7a06c58-5567-40aa-b24e-c5f71943f570\") " pod="openshift-insights/insights-runtime-extractor-bggdc"
Jul 07 15:45:03.288862 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:03.288839    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b7a06c58-5567-40aa-b24e-c5f71943f570-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bggdc\" (UID: \"b7a06c58-5567-40aa-b24e-c5f71943f570\") " pod="openshift-insights/insights-runtime-extractor-bggdc"
Jul 07 15:45:03.345051 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:03.345014    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bggdc"
Jul 07 15:45:03.470921 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:03.470872    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bggdc"]
Jul 07 15:45:03.473964 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:45:03.473929    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7a06c58_5567_40aa_b24e_c5f71943f570.slice/crio-e446960f36508fa0469dd4186b251e77d8bd8442cbf063358d2e5760857b9d27 WatchSource:0}: Error finding container e446960f36508fa0469dd4186b251e77d8bd8442cbf063358d2e5760857b9d27: Status 404 returned error can't find the container with id e446960f36508fa0469dd4186b251e77d8bd8442cbf063358d2e5760857b9d27
Jul 07 15:45:03.743634 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:03.743591    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bggdc" event={"ID":"b7a06c58-5567-40aa-b24e-c5f71943f570","Type":"ContainerStarted","Data":"e446960f36508fa0469dd4186b251e77d8bd8442cbf063358d2e5760857b9d27"}
Jul 07 15:45:07.674207 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:07.674178    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vfd6r"
Jul 07 15:45:07.759635 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:07.759601    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bggdc" event={"ID":"b7a06c58-5567-40aa-b24e-c5f71943f570","Type":"ContainerStarted","Data":"d4ba02ddab6d234760a93db1171b42dbe27b3c2a19d05bd67395ddd1244db427"}
Jul 07 15:45:08.764797 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:08.764764    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bggdc" event={"ID":"b7a06c58-5567-40aa-b24e-c5f71943f570","Type":"ContainerStarted","Data":"5e6be9678dda3d4850347d3cc6693a097ac7fd84871ad1daa4e756187ea95a50"}
Jul 07 15:45:11.739089 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:11.739057    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-z47rg"
Jul 07 15:45:11.776292 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:11.776255    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bggdc" event={"ID":"b7a06c58-5567-40aa-b24e-c5f71943f570","Type":"ContainerStarted","Data":"8106d6c795deafbad79fa2ee337bc028003a2752322087e0c1e3f7945f799441"}
Jul 07 15:45:11.796273 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:11.796117    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-bggdc" podStartSLOduration=17.474711762 podStartE2EDuration="24.796097879s" totalImagesPullingTime="7.321386117s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 15:44:47 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:45:11.795651706 +0000 UTC m=+65.884077389" watchObservedRunningTime="2026-07-07 15:45:11.796097879 +0000 UTC m=+65.884523548"
Jul 07 15:45:12.162852 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:12.162755    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a773f68-4a73-415d-abc9-3672aa14a254-metrics-certs\") pod \"network-metrics-daemon-zbhgc\" (UID: \"2a773f68-4a73-415d-abc9-3672aa14a254\") " pod="openshift-multus/network-metrics-daemon-zbhgc"
Jul 07 15:45:12.162852 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:12.162794    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f0d07c9b-7fac-4665-b397-d996f6781c50-original-pull-secret\") pod \"global-pull-secret-syncer-fvkrg\" (UID: \"f0d07c9b-7fac-4665-b397-d996f6781c50\") " pod="kube-system/global-pull-secret-syncer-fvkrg"
Jul 07 15:45:12.165469 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:12.165451    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 07 15:45:12.165469 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:12.165465    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 07 15:45:12.175509 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:12.175474    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f0d07c9b-7fac-4665-b397-d996f6781c50-original-pull-secret\") pod \"global-pull-secret-syncer-fvkrg\" (UID: \"f0d07c9b-7fac-4665-b397-d996f6781c50\") " pod="kube-system/global-pull-secret-syncer-fvkrg"
Jul 07 15:45:12.175623 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:12.175553    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a773f68-4a73-415d-abc9-3672aa14a254-metrics-certs\") pod \"network-metrics-daemon-zbhgc\" (UID: \"2a773f68-4a73-415d-abc9-3672aa14a254\") " pod="openshift-multus/network-metrics-daemon-zbhgc"
Jul 07 15:45:12.264157 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:12.264104    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4wkf\" (UniqueName: \"kubernetes.io/projected/e86522b9-3dc8-41c0-8d8a-40229b4795a7-kube-api-access-m4wkf\") pod \"network-check-target-tx5v9\" (UID: \"e86522b9-3dc8-41c0-8d8a-40229b4795a7\") " pod="openshift-network-diagnostics/network-check-target-tx5v9"
Jul 07 15:45:12.266664 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:12.266646    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 07 15:45:12.277218 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:12.277197    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 07 15:45:12.280165 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:12.280150    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-wcvdn\""
Jul 07 15:45:12.287466 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:12.287440    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4wkf\" (UniqueName: \"kubernetes.io/projected/e86522b9-3dc8-41c0-8d8a-40229b4795a7-kube-api-access-m4wkf\") pod \"network-check-target-tx5v9\" (UID: \"e86522b9-3dc8-41c0-8d8a-40229b4795a7\") " pod="openshift-network-diagnostics/network-check-target-tx5v9"
Jul 07 15:45:12.288992 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:12.288976    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zbhgc"
Jul 07 15:45:12.290197 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:12.290180    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fvkrg"
Jul 07 15:45:12.424578 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:12.424496    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zbhgc"]
Jul 07 15:45:12.427448 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:45:12.427424    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a773f68_4a73_415d_abc9_3672aa14a254.slice/crio-b15f2a99afe1e1c8a8a8d75e210cab72cff500df67c4b39fcaa30a90001492e9 WatchSource:0}: Error finding container b15f2a99afe1e1c8a8a8d75e210cab72cff500df67c4b39fcaa30a90001492e9: Status 404 returned error can't find the container with id b15f2a99afe1e1c8a8a8d75e210cab72cff500df67c4b39fcaa30a90001492e9
Jul 07 15:45:12.438102 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:12.438064    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-fvkrg"]
Jul 07 15:45:12.442003 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:45:12.441965    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0d07c9b_7fac_4665_b397_d996f6781c50.slice/crio-90beb9246696c2b03aef9bc4618a9be03692dcfa70e18564e0e9c0fcea54eba7 WatchSource:0}: Error finding container 90beb9246696c2b03aef9bc4618a9be03692dcfa70e18564e0e9c0fcea54eba7: Status 404 returned error can't find the container with id 90beb9246696c2b03aef9bc4618a9be03692dcfa70e18564e0e9c0fcea54eba7
Jul 07 15:45:12.587790 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:12.587762    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ctzps\""
Jul 07 15:45:12.595359 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:12.595336    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tx5v9"
Jul 07 15:45:12.721080 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:12.721050    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tx5v9"]
Jul 07 15:45:12.724133 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:45:12.724105    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode86522b9_3dc8_41c0_8d8a_40229b4795a7.slice/crio-c9f6f8a59cc4030b7f90895ec069ea3cf3417cad5aeae50639176b26535829e9 WatchSource:0}: Error finding container c9f6f8a59cc4030b7f90895ec069ea3cf3417cad5aeae50639176b26535829e9: Status 404 returned error can't find the container with id c9f6f8a59cc4030b7f90895ec069ea3cf3417cad5aeae50639176b26535829e9
Jul 07 15:45:12.780953 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:12.780913    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-fvkrg" event={"ID":"f0d07c9b-7fac-4665-b397-d996f6781c50","Type":"ContainerStarted","Data":"90beb9246696c2b03aef9bc4618a9be03692dcfa70e18564e0e9c0fcea54eba7"}
Jul 07 15:45:12.782082 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:12.782058    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tx5v9" event={"ID":"e86522b9-3dc8-41c0-8d8a-40229b4795a7","Type":"ContainerStarted","Data":"c9f6f8a59cc4030b7f90895ec069ea3cf3417cad5aeae50639176b26535829e9"}
Jul 07 15:45:12.783204 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:12.783180    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zbhgc" event={"ID":"2a773f68-4a73-415d-abc9-3672aa14a254","Type":"ContainerStarted","Data":"b15f2a99afe1e1c8a8a8d75e210cab72cff500df67c4b39fcaa30a90001492e9"}
Jul 07 15:45:13.358475 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:13.358443    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f9998b7cf-8w24j"]
Jul 07 15:45:13.361557 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:13.360764    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f9998b7cf-8w24j"
Jul 07 15:45:13.370297 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:13.370272    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 07 15:45:13.370914 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:13.370544    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 07 15:45:13.370914 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:13.370765    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 07 15:45:13.371394 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:13.371184    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 07 15:45:13.388237 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:13.388109    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6567dcbfb8-xb7hb"]
Jul 07 15:45:13.390395 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:13.390368    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-6qpwk"]
Jul 07 15:45:13.391082 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:13.391063    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6567dcbfb8-xb7hb"
Jul 07 15:45:13.392922 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:13.392901    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bf4bc488b-q22mp"]
Jul 07 15:45:13.393032 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:13.393016    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-6qpwk"
Jul 07 15:45:13.393599 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:13.393578    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-5g92z\""
Jul 07 15:45:13.394507 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:13.394485    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 07 15:45:13.395181 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:13.395161    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bf4bc488b-q22mp"
Jul 07 15:45:13.396385 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:13.395075    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f9998b7cf-8w24j"]
Jul 07 15:45:13.397296 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:13.397198    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 07 15:45:13.397392 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:13.397204    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 07 15:45:13.399227 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:13.399207    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 07 15:45:13.399453 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:13.399427    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 07 15:45:13.399923 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:13.399648    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 07 15:45:13.399923 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:13.399693    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7c68df6494-4mrwk"]
Jul 07 15:45:13.400065 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:13.400044    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-zzbt5\""
Jul 07 15:45:13.400479 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:13.400457    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 07 15:45:13.408592 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:13.408458    2566 patch_prober.go:28] interesting pod/image-registry-7c68df6494-4mrwk container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 07 15:45:13.408592 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:13.408522    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-7c68df6494-4mrwk" podUID="54bc412f-9e52-45a2-b820-3dee0d9902d7" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 15:45:13.411288 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:13.411257    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6567dcbfb8-xb7hb"]
Jul 07 15:45:13.414202 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:13.414175    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-6qpwk"]
Jul 07 15:45:13.424732 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:13.424705    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bf4bc488b-q22mp"]
Jul 07 15:45:13.473522 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:13.473485    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3538f866-cd99-4f24-ad38-fa444a4fdb05-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5bf4bc488b-q22mp\" (UID: \"3538f866-cd99-4f24-ad38-fa444a4fdb05\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bf4bc488b-q22mp"
Jul 07 15:45:13.473805 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:13.473550    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/3538f866-cd99-4f24-ad38-fa444a4fdb05-hub\") pod \"cluster-proxy-proxy-agent-5bf4bc488b-q22mp\" (UID: \"3538f866-cd99-4f24-ad38-fa444a4fdb05\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bf4bc488b-q22mp"
Jul 07 15:45:13.473805 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:13.473576    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/3538f866-cd99-4f24-ad38-fa444a4fdb05-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5bf4bc488b-q22mp\" (UID: \"3538f866-cd99-4f24-ad38-fa444a4fdb05\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bf4bc488b-q22mp"
Jul 07 15:45:13.473805 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:13.473603    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r67w6\" (UniqueName: \"kubernetes.io/projected/8c3fa5f2-200d-4b2c-adb8-5eb2a5ab9385-kube-api-access-r67w6\") pod \"klusterlet-addon-workmgr-6f9998b7cf-8w24j\" (UID: \"8c3fa5f2-200d-4b2c-adb8-5eb2a5ab9385\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f9998b7cf-8w24j"
Jul 07 15:45:13.473805 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:13.473627    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/3538f866-cd99-4f24-ad38-fa444a4fdb05-ca\") pod \"cluster-proxy-proxy-agent-5bf4bc488b-q22mp\" (UID: \"3538f866-cd99-4f24-ad38-fa444a4fdb05\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bf4bc488b-q22mp"
Jul 07 15:45:13.473805 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:13.473662    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d66b1fe3-70c4-422d-b909-9555280a9d1c-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-6qpwk\" (UID: \"d66b1fe3-70c4-422d-b909-9555280a9d1c\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-6qpwk"
Jul 07 15:45:13.473805 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:13.473746    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/3538f866-cd99-4f24-ad38-fa444a4fdb05-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5bf4bc488b-q22mp\" (UID: \"3538f866-cd99-4f24-ad38-fa444a4fdb05\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bf4bc488b-q22mp"
Jul 07 15:45:13.474169 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:13.473825    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/106ff6c4-e1d4-4984-b495-65773809eff9-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6567dcbfb8-xb7hb\" (UID: \"106ff6c4-e1d4-4984-b495-65773809eff9\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6567dcbfb8-xb7hb"
Jul 07 15:45:13.474169 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:13.473869    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggl6g\" (UniqueName: \"kubernetes.io/projected/106ff6c4-e1d4-4984-b495-65773809eff9-kube-api-access-ggl6g\") pod \"managed-serviceaccount-addon-agent-6567dcbfb8-xb7hb\" (UID: \"106ff6c4-e1d4-4984-b495-65773809eff9\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6567dcbfb8-xb7hb"
Jul 07 15:45:13.474169 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:13.473917    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d66b1fe3-70c4-422d-b909-9555280a9d1c-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-6qpwk\" (UID: \"d66b1fe3-70c4-422d-b909-9555280a9d1c\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-6qpwk"
Jul 07 15:45:13.474169 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:13.473946    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tvkl\" (UniqueName: \"kubernetes.io/projected/3538f866-cd99-4f24-ad38-fa444a4fdb05-kube-api-access-7tvkl\") pod \"cluster-proxy-proxy-agent-5bf4bc488b-q22mp\" (UID: \"3538f866-cd99-4f24-ad38-fa444a4fdb05\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bf4bc488b-q22mp"
Jul 07 15:45:13.474169 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:13.473981    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8c3fa5f2-200d-4b2c-adb8-5eb2a5ab9385-tmp\") pod \"klusterlet-addon-workmgr-6f9998b7cf-8w24j\" (UID: \"8c3fa5f2-200d-4b2c-adb8-5eb2a5ab9385\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f9998b7cf-8w24j"
Jul 07 15:45:13.474169 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:13.474032    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/8c3fa5f2-200d-4b2c-adb8-5eb2a5ab9385-klusterlet-config\") pod \"klusterlet-addon-workmgr-6f9998b7cf-8w24j\" (UID: \"8c3fa5f2-200d-4b2c-adb8-5eb2a5ab9385\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f9998b7cf-8w24j"
Jul 07 15:45:13.489364 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:13.489333    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-77d7b5bb7d-rcmxv"]
Jul 07 15:45:13.492568 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:13.492543    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-77d7b5bb7d-rcmxv"
Jul 07 15:45:13.514518 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:13.514400    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-77d7b5bb7d-rcmxv"]
Jul 07 15:45:13.575191 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:13.575153    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3538f866-cd99-4f24-ad38-fa444a4fdb05-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5bf4bc488b-q22mp\" (UID: \"3538f866-cd99-4f24-ad38-fa444a4fdb05\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bf4bc488b-q22mp"
Jul 07 15:45:13.575364 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:13.575211    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/760703f9-9ff6-4867-bef4-85d0cb8c0687-registry-certificates\") pod \"image-registry-77d7b5bb7d-rcmxv\" (UID: \"760703f9-9ff6-4867-bef4-85d0cb8c0687\") " pod="openshift-image-registry/image-registry-77d7b5bb7d-rcmxv"
Jul 07 15:45:13.575364 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:13.575241    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/760703f9-9ff6-4867-bef4-85d0cb8c0687-installation-pull-secrets\") pod \"image-registry-77d7b5bb7d-rcmxv\" (UID: \"760703f9-9ff6-4867-bef4-85d0cb8c0687\") " pod="openshift-image-registry/image-registry-77d7b5bb7d-rcmxv"
Jul 07 15:45:13.575364 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:13.575309    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/3538f866-cd99-4f24-ad38-fa444a4fdb05-hub\") pod \"cluster-proxy-proxy-agent-5bf4bc488b-q22mp\" (UID: \"3538f866-cd99-4f24-ad38-fa444a4fdb05\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bf4bc488b-q22mp"
Jul 07 15:45:13.575364 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:13.575346    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/3538f866-cd99-4f24-ad38-fa444a4fdb05-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5bf4bc488b-q22mp\" (UID: \"3538f866-cd99-4f24-ad38-fa444a4fdb05\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bf4bc488b-q22mp"
Jul 07 15:45:13.575900 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:13.575385    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/760703f9-9ff6-4867-bef4-85d0cb8c0687-trusted-ca\") pod \"image-registry-77d7b5bb7d-rcmxv\" (UID: \"760703f9-9ff6-4867-bef4-85d0cb8c0687\") " pod="openshift-image-registry/image-registry-77d7b5bb7d-rcmxv"
Jul 07 15:45:13.575900 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:13.575421    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r67w6\" (UniqueName: \"kubernetes.io/projected/8c3fa5f2-200d-4b2c-adb8-5eb2a5ab9385-kube-api-access-r67w6\") pod \"klusterlet-addon-workmgr-6f9998b7cf-8w24j\" (UID: \"8c3fa5f2-200d-4b2c-adb8-5eb2a5ab9385\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f9998b7cf-8w24j"
Jul 07 15:45:13.575900 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:13.575447    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/3538f866-cd99-4f24-ad38-fa444a4fdb05-ca\") pod \"cluster-proxy-proxy-agent-5bf4bc488b-q22mp\" (UID: \"3538f866-cd99-4f24-ad38-fa444a4fdb05\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bf4bc488b-q22mp"
Jul 07 15:45:13.575900 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:13.575485    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d66b1fe3-70c4-422d-b909-9555280a9d1c-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-6qpwk\" (UID: \"d66b1fe3-70c4-422d-b909-9555280a9d1c\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-6qpwk"
Jul 07 15:45:13.575900 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:13.575525    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/3538f866-cd99-4f24-ad38-fa444a4fdb05-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5bf4bc488b-q22mp\" (UID: \"3538f866-cd99-4f24-ad38-fa444a4fdb05\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bf4bc488b-q22mp"
Jul 07 15:45:13.575900 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:13.575553    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/106ff6c4-e1d4-4984-b495-65773809eff9-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6567dcbfb8-xb7hb\" (UID: \"106ff6c4-e1d4-4984-b495-65773809eff9\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6567dcbfb8-xb7hb"
Jul 07 15:45:13.575900 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:13.575602    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggl6g\" (UniqueName: \"kubernetes.io/projected/106ff6c4-e1d4-4984-b495-65773809eff9-kube-api-access-ggl6g\") pod \"managed-serviceaccount-addon-agent-6567dcbfb8-xb7hb\" (UID: \"106ff6c4-e1d4-4984-b495-65773809eff9\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6567dcbfb8-xb7hb"
Jul 07 15:45:13.575900 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:13.575629    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d66b1fe3-70c4-422d-b909-9555280a9d1c-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-6qpwk\" (UID: \"d66b1fe3-70c4-422d-b909-9555280a9d1c\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-6qpwk"
Jul 07 15:45:13.575900 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:13.575656    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/760703f9-9ff6-4867-bef4-85d0cb8c0687-ca-trust-extracted\") pod \"image-registry-77d7b5bb7d-rcmxv\" (UID: \"760703f9-9ff6-4867-bef4-85d0cb8c0687\") " pod="openshift-image-registry/image-registry-77d7b5bb7d-rcmxv"
Jul 07 15:45:13.575900 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:13.575684    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tvkl\" (UniqueName: \"kubernetes.io/projected/3538f866-cd99-4f24-ad38-fa444a4fdb05-kube-api-access-7tvkl\") pod \"cluster-proxy-proxy-agent-5bf4bc488b-q22mp\" (UID: \"3538f866-cd99-4f24-ad38-fa444a4fdb05\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bf4bc488b-q22mp"
Jul 07 15:45:13.575900 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:13.575713    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pvvc\" (UniqueName: \"kubernetes.io/projected/760703f9-9ff6-4867-bef4-85d0cb8c0687-kube-api-access-8pvvc\") pod \"image-registry-77d7b5bb7d-rcmxv\" (UID: \"760703f9-9ff6-4867-bef4-85d0cb8c0687\") " pod="openshift-image-registry/image-registry-77d7b5bb7d-rcmxv"
Jul 07 15:45:13.575900 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:13.575747    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8c3fa5f2-200d-4b2c-adb8-5eb2a5ab9385-tmp\") pod \"klusterlet-addon-workmgr-6f9998b7cf-8w24j\" (UID: \"8c3fa5f2-200d-4b2c-adb8-5eb2a5ab9385\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f9998b7cf-8w24j"
Jul 07 15:45:13.575900 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:13.575772    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/760703f9-9ff6-4867-bef4-85d0cb8c0687-image-registry-private-configuration\") pod \"image-registry-77d7b5bb7d-rcmxv\" (UID: \"760703f9-9ff6-4867-bef4-85d0cb8c0687\") " pod="openshift-image-registry/image-registry-77d7b5bb7d-rcmxv"
Jul 07 15:45:13.575900 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:13.575825    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/760703f9-9ff6-4867-bef4-85d0cb8c0687-registry-tls\") pod \"image-registry-77d7b5bb7d-rcmxv\" (UID: \"760703f9-9ff6-4867-bef4-85d0cb8c0687\") " pod="openshift-image-registry/image-registry-77d7b5bb7d-rcmxv"
Jul 07 15:45:13.575900 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:13.575875    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/760703f9-9ff6-4867-bef4-85d0cb8c0687-bound-sa-token\") pod \"image-registry-77d7b5bb7d-rcmxv\" (UID: \"760703f9-9ff6-4867-bef4-85d0cb8c0687\") " pod="openshift-image-registry/image-registry-77d7b5bb7d-rcmxv"
Jul 07 15:45:13.577179 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:13.575942    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/8c3fa5f2-200d-4b2c-adb8-5eb2a5ab9385-klusterlet-config\") pod \"klusterlet-addon-workmgr-6f9998b7cf-8w24j\" (UID: \"8c3fa5f2-200d-4b2c-adb8-5eb2a5ab9385\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f9998b7cf-8w24j"
Jul 07 15:45:13.577179 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:13.576126    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/3538f866-cd99-4f24-ad38-fa444a4fdb05-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5bf4bc488b-q22mp\" (UID: \"3538f866-cd99-4f24-ad38-fa444a4fdb05\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bf4bc488b-q22mp"
Jul 07 15:45:13.577179 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:13.576323    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8c3fa5f2-200d-4b2c-adb8-5eb2a5ab9385-tmp\") pod \"klusterlet-addon-workmgr-6f9998b7cf-8w24j\" (UID: \"8c3fa5f2-200d-4b2c-adb8-5eb2a5ab9385\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f9998b7cf-8w24j"
Jul 07 15:45:13.577347 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:13.577248    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d66b1fe3-70c4-422d-b909-9555280a9d1c-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-6qpwk\" (UID: \"d66b1fe3-70c4-422d-b909-9555280a9d1c\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-6qpwk"
Jul 07 15:45:13.581366 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:13.581316    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/106ff6c4-e1d4-4984-b495-65773809eff9-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6567dcbfb8-xb7hb\" (UID: \"106ff6c4-e1d4-4984-b495-65773809eff9\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6567dcbfb8-xb7hb"
Jul 07 15:45:13.582068 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:13.581992    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/8c3fa5f2-200d-4b2c-adb8-5eb2a5ab9385-klusterlet-config\") pod \"klusterlet-addon-workmgr-6f9998b7cf-8w24j\" (UID: \"8c3fa5f2-200d-4b2c-adb8-5eb2a5ab9385\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f9998b7cf-8w24j"
Jul 07 15:45:13.582842 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:13.582795    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/3538f866-cd99-4f24-ad38-fa444a4fdb05-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5bf4bc488b-q22mp\" (UID: \"3538f866-cd99-4f24-ad38-fa444a4fdb05\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bf4bc488b-q22mp"
Jul 07 15:45:13.583469 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:13.583368    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/3538f866-cd99-4f24-ad38-fa444a4fdb05-ca\") pod \"cluster-proxy-proxy-agent-5bf4bc488b-q22mp\" (UID: \"3538f866-cd99-4f24-ad38-fa444a4fdb05\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bf4bc488b-q22mp"
Jul 07 15:45:13.583469 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:13.583430    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/3538f866-cd99-4f24-ad38-fa444a4fdb05-hub\") pod \"cluster-proxy-proxy-agent-5bf4bc488b-q22mp\" (UID: \"3538f866-cd99-4f24-ad38-fa444a4fdb05\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bf4bc488b-q22mp"
Jul 07 15:45:13.584230 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:13.584206    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3538f866-cd99-4f24-ad38-fa444a4fdb05-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5bf4bc488b-q22mp\" (UID: \"3538f866-cd99-4f24-ad38-fa444a4fdb05\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bf4bc488b-q22mp"
Jul 07 15:45:13.585940 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:13.585877    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d66b1fe3-70c4-422d-b909-9555280a9d1c-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-6qpwk\" (UID: \"d66b1fe3-70c4-422d-b909-9555280a9d1c\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-6qpwk"
Jul 07 15:45:13.600993 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:13.600844    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r67w6\" (UniqueName: \"kubernetes.io/projected/8c3fa5f2-200d-4b2c-adb8-5eb2a5ab9385-kube-api-access-r67w6\") pod \"klusterlet-addon-workmgr-6f9998b7cf-8w24j\" (UID: \"8c3fa5f2-200d-4b2c-adb8-5eb2a5ab9385\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f9998b7cf-8w24j"
Jul 07 15:45:13.603454 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:13.603384    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggl6g\" (UniqueName: \"kubernetes.io/projected/106ff6c4-e1d4-4984-b495-65773809eff9-kube-api-access-ggl6g\") pod \"managed-serviceaccount-addon-agent-6567dcbfb8-xb7hb\" (UID: \"106ff6c4-e1d4-4984-b495-65773809eff9\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6567dcbfb8-xb7hb"
Jul 07 15:45:13.603566 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:13.603487    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tvkl\" (UniqueName: \"kubernetes.io/projected/3538f866-cd99-4f24-ad38-fa444a4fdb05-kube-api-access-7tvkl\") pod \"cluster-proxy-proxy-agent-5bf4bc488b-q22mp\" (UID: \"3538f866-cd99-4f24-ad38-fa444a4fdb05\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bf4bc488b-q22mp"
Jul 07 15:45:13.677239 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:13.677156    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f9998b7cf-8w24j"
Jul 07 15:45:13.677401 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:13.677281    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/760703f9-9ff6-4867-bef4-85d0cb8c0687-ca-trust-extracted\") pod \"image-registry-77d7b5bb7d-rcmxv\" (UID: \"760703f9-9ff6-4867-bef4-85d0cb8c0687\") " pod="openshift-image-registry/image-registry-77d7b5bb7d-rcmxv"
Jul 07 15:45:13.677401 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:13.677340    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pvvc\" (UniqueName: \"kubernetes.io/projected/760703f9-9ff6-4867-bef4-85d0cb8c0687-kube-api-access-8pvvc\") pod \"image-registry-77d7b5bb7d-rcmxv\" (UID: \"760703f9-9ff6-4867-bef4-85d0cb8c0687\") " pod="openshift-image-registry/image-registry-77d7b5bb7d-rcmxv"
Jul 07 15:45:13.677401 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:13.677377    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/760703f9-9ff6-4867-bef4-85d0cb8c0687-image-registry-private-configuration\") pod \"image-registry-77d7b5bb7d-rcmxv\" (UID: \"760703f9-9ff6-4867-bef4-85d0cb8c0687\") " pod="openshift-image-registry/image-registry-77d7b5bb7d-rcmxv"
Jul 07 15:45:13.677632 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:13.677408    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/760703f9-9ff6-4867-bef4-85d0cb8c0687-registry-tls\") pod \"image-registry-77d7b5bb7d-rcmxv\" (UID: \"760703f9-9ff6-4867-bef4-85d0cb8c0687\") " pod="openshift-image-registry/image-registry-77d7b5bb7d-rcmxv"
Jul 07 15:45:13.677632 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:13.677451    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/760703f9-9ff6-4867-bef4-85d0cb8c0687-bound-sa-token\") pod \"image-registry-77d7b5bb7d-rcmxv\" (UID: \"760703f9-9ff6-4867-bef4-85d0cb8c0687\") " pod="openshift-image-registry/image-registry-77d7b5bb7d-rcmxv"
Jul 07 15:45:13.677632 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:13.677507    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/760703f9-9ff6-4867-bef4-85d0cb8c0687-registry-certificates\") pod \"image-registry-77d7b5bb7d-rcmxv\" (UID: \"760703f9-9ff6-4867-bef4-85d0cb8c0687\") " pod="openshift-image-registry/image-registry-77d7b5bb7d-rcmxv"
Jul 07 15:45:13.677632 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:13.677536    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/760703f9-9ff6-4867-bef4-85d0cb8c0687-installation-pull-secrets\") pod \"image-registry-77d7b5bb7d-rcmxv\" (UID: \"760703f9-9ff6-4867-bef4-85d0cb8c0687\") " pod="openshift-image-registry/image-registry-77d7b5bb7d-rcmxv"
Jul 07 15:45:13.677632 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:13.677569    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/760703f9-9ff6-4867-bef4-85d0cb8c0687-trusted-ca\") pod \"image-registry-77d7b5bb7d-rcmxv\" (UID: \"760703f9-9ff6-4867-bef4-85d0cb8c0687\") " pod="openshift-image-registry/image-registry-77d7b5bb7d-rcmxv"
Jul 07 15:45:13.677915 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:13.677761    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/760703f9-9ff6-4867-bef4-85d0cb8c0687-ca-trust-extracted\") pod \"image-registry-77d7b5bb7d-rcmxv\" (UID: \"760703f9-9ff6-4867-bef4-85d0cb8c0687\") " pod="openshift-image-registry/image-registry-77d7b5bb7d-rcmxv"
Jul 07 15:45:13.679306 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:13.679224    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/760703f9-9ff6-4867-bef4-85d0cb8c0687-registry-certificates\") pod \"image-registry-77d7b5bb7d-rcmxv\" (UID: \"760703f9-9ff6-4867-bef4-85d0cb8c0687\") " pod="openshift-image-registry/image-registry-77d7b5bb7d-rcmxv"
Jul 07 15:45:13.680245 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:13.680087    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/760703f9-9ff6-4867-bef4-85d0cb8c0687-trusted-ca\") pod \"image-registry-77d7b5bb7d-rcmxv\" (UID: \"760703f9-9ff6-4867-bef4-85d0cb8c0687\") " pod="openshift-image-registry/image-registry-77d7b5bb7d-rcmxv"
Jul 07 15:45:13.686076 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:13.681245    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/760703f9-9ff6-4867-bef4-85d0cb8c0687-image-registry-private-configuration\") pod \"image-registry-77d7b5bb7d-rcmxv\" (UID: \"760703f9-9ff6-4867-bef4-85d0cb8c0687\") " pod="openshift-image-registry/image-registry-77d7b5bb7d-rcmxv"
Jul 07 15:45:13.686076 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:13.681580    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/760703f9-9ff6-4867-bef4-85d0cb8c0687-registry-tls\") pod \"image-registry-77d7b5bb7d-rcmxv\" (UID: \"760703f9-9ff6-4867-bef4-85d0cb8c0687\") " pod="openshift-image-registry/image-registry-77d7b5bb7d-rcmxv"
Jul 07 15:45:13.686076 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:13.683397    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/760703f9-9ff6-4867-bef4-85d0cb8c0687-installation-pull-secrets\") pod \"image-registry-77d7b5bb7d-rcmxv\" (UID: \"760703f9-9ff6-4867-bef4-85d0cb8c0687\") " pod="openshift-image-registry/image-registry-77d7b5bb7d-rcmxv"
Jul 07 15:45:13.697961 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:13.697899    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pvvc\" (UniqueName: \"kubernetes.io/projected/760703f9-9ff6-4867-bef4-85d0cb8c0687-kube-api-access-8pvvc\") pod \"image-registry-77d7b5bb7d-rcmxv\" (UID: \"760703f9-9ff6-4867-bef4-85d0cb8c0687\") " pod="openshift-image-registry/image-registry-77d7b5bb7d-rcmxv"
Jul 07 15:45:13.720391 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:13.719614    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6567dcbfb8-xb7hb"
Jul 07 15:45:13.723472 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:13.723405    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/760703f9-9ff6-4867-bef4-85d0cb8c0687-bound-sa-token\") pod \"image-registry-77d7b5bb7d-rcmxv\" (UID: \"760703f9-9ff6-4867-bef4-85d0cb8c0687\") " pod="openshift-image-registry/image-registry-77d7b5bb7d-rcmxv"
Jul 07 15:45:13.731234 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:13.731202    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-6qpwk"
Jul 07 15:45:13.739667 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:13.739639    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bf4bc488b-q22mp"
Jul 07 15:45:13.791088 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:13.790972    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zbhgc" event={"ID":"2a773f68-4a73-415d-abc9-3672aa14a254","Type":"ContainerStarted","Data":"398720a235e928c8dd458f121bc14d41925290a5c6c7dd3da575ec939378a0ff"}
Jul 07 15:45:13.816686 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:13.816353    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-77d7b5bb7d-rcmxv"
Jul 07 15:45:13.899949 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:13.899852    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f9998b7cf-8w24j"]
Jul 07 15:45:13.905348 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:45:13.905299    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c3fa5f2_200d_4b2c_adb8_5eb2a5ab9385.slice/crio-6ecdde109937f3bcff5c49fb96c22d2d36616da13597450663fc56d44ef67d3a WatchSource:0}: Error finding container 6ecdde109937f3bcff5c49fb96c22d2d36616da13597450663fc56d44ef67d3a: Status 404 returned error can't find the container with id 6ecdde109937f3bcff5c49fb96c22d2d36616da13597450663fc56d44ef67d3a
Jul 07 15:45:13.934478 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:13.934408    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6567dcbfb8-xb7hb"]
Jul 07 15:45:13.987560 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:13.987477    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-6qpwk"]
Jul 07 15:45:13.992388 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:45:13.992344    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd66b1fe3_70c4_422d_b909_9555280a9d1c.slice/crio-cae8825eb2d6dbb718c1c68608387f6ce70e7241b3cb931007359af8604488b7 WatchSource:0}: Error finding container cae8825eb2d6dbb718c1c68608387f6ce70e7241b3cb931007359af8604488b7: Status 404 returned error can't find the container with id cae8825eb2d6dbb718c1c68608387f6ce70e7241b3cb931007359af8604488b7
Jul 07 15:45:14.003156 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:14.003106    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bf4bc488b-q22mp"]
Jul 07 15:45:14.005223 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:45:14.005143    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3538f866_cd99_4f24_ad38_fa444a4fdb05.slice/crio-53e3edcc8b1af0961cda5921ab3d0a8d56fb0a39bd3c0d34264ba3d3b8a9fa41 WatchSource:0}: Error finding container 53e3edcc8b1af0961cda5921ab3d0a8d56fb0a39bd3c0d34264ba3d3b8a9fa41: Status 404 returned error can't find the container with id 53e3edcc8b1af0961cda5921ab3d0a8d56fb0a39bd3c0d34264ba3d3b8a9fa41
Jul 07 15:45:14.050386 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:14.050352    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-77d7b5bb7d-rcmxv"]
Jul 07 15:45:14.053135 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:45:14.053104    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod760703f9_9ff6_4867_bef4_85d0cb8c0687.slice/crio-1eaa60b053190a4fb7c043c05a23f1304bf529fbd31741036171f954f06dbee6 WatchSource:0}: Error finding container 1eaa60b053190a4fb7c043c05a23f1304bf529fbd31741036171f954f06dbee6: Status 404 returned error can't find the container with id 1eaa60b053190a4fb7c043c05a23f1304bf529fbd31741036171f954f06dbee6
Jul 07 15:45:14.796975 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:14.796929    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-77d7b5bb7d-rcmxv" event={"ID":"760703f9-9ff6-4867-bef4-85d0cb8c0687","Type":"ContainerStarted","Data":"43379bf46043ac0954ce72e22c2b9b1b2ba7843d8e20f3095710121d3be7eb0e"}
Jul 07 15:45:14.797453 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:14.796996    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-77d7b5bb7d-rcmxv" event={"ID":"760703f9-9ff6-4867-bef4-85d0cb8c0687","Type":"ContainerStarted","Data":"1eaa60b053190a4fb7c043c05a23f1304bf529fbd31741036171f954f06dbee6"}
Jul 07 15:45:14.797749 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:14.797727    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-77d7b5bb7d-rcmxv"
Jul 07 15:45:14.799850 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:14.799823    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bf4bc488b-q22mp" event={"ID":"3538f866-cd99-4f24-ad38-fa444a4fdb05","Type":"ContainerStarted","Data":"53e3edcc8b1af0961cda5921ab3d0a8d56fb0a39bd3c0d34264ba3d3b8a9fa41"}
Jul 07 15:45:14.801299 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:14.801271    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6567dcbfb8-xb7hb" event={"ID":"106ff6c4-e1d4-4984-b495-65773809eff9","Type":"ContainerStarted","Data":"f45663f162ba6e87d19c17e9bea7782b5b591c91824671850ac2724d933632a0"}
Jul 07 15:45:14.804083 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:14.804056    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zbhgc" event={"ID":"2a773f68-4a73-415d-abc9-3672aa14a254","Type":"ContainerStarted","Data":"b674281c7845f10302fec099070d623f5d3eb60b1a0107b97cba9b8f0fed0ffa"}
Jul 07 15:45:14.805880 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:14.805805    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6bffdf9778-6qpwk" event={"ID":"d66b1fe3-70c4-422d-b909-9555280a9d1c","Type":"ContainerStarted","Data":"cae8825eb2d6dbb718c1c68608387f6ce70e7241b3cb931007359af8604488b7"}
Jul 07 15:45:14.807139 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:14.807089    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f9998b7cf-8w24j" event={"ID":"8c3fa5f2-200d-4b2c-adb8-5eb2a5ab9385","Type":"ContainerStarted","Data":"6ecdde109937f3bcff5c49fb96c22d2d36616da13597450663fc56d44ef67d3a"}
Jul 07 15:45:14.825068 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:14.824638    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-77d7b5bb7d-rcmxv" podStartSLOduration=1.8246144819999999 podStartE2EDuration="1.824614482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:45:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:45:14.821930525 +0000 UTC m=+68.910356196" watchObservedRunningTime="2026-07-07 15:45:14.824614482 +0000 UTC m=+68.913040153"
Jul 07 15:45:14.843162 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:14.843037    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-zbhgc" podStartSLOduration=66.738331458 podStartE2EDuration="1m7.843011897s" totalImagesPullingTime="1.104680439s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:44:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:45:14.841755239 +0000 UTC m=+68.930180903" watchObservedRunningTime="2026-07-07 15:45:14.843011897 +0000 UTC m=+68.931437579"
Jul 07 15:45:23.405093 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:23.405019    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7c68df6494-4mrwk"
Jul 07 15:45:23.847034 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:23.846992    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-fvkrg" event={"ID":"f0d07c9b-7fac-4665-b397-d996f6781c50","Type":"ContainerStarted","Data":"5897b61c97114f22ad7c966f7e3aebb2089625e785ac8f8616dba71b8aad5303"}
Jul 07 15:45:23.848404 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:23.848379    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tx5v9" event={"ID":"e86522b9-3dc8-41c0-8d8a-40229b4795a7","Type":"ContainerStarted","Data":"46e0f84a124aaa83cbbc0b942307fea1c74a3b109911bdbe4bdea106afa59ed4"}
Jul 07 15:45:23.848523 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:23.848432    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-tx5v9"
Jul 07 15:45:23.850386 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:23.850364    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bf4bc488b-q22mp" event={"ID":"3538f866-cd99-4f24-ad38-fa444a4fdb05","Type":"ContainerStarted","Data":"e4c84ccfb992a6484b030e2f5555def6356b47dba08ebbaec558058e677ea67c"}
Jul 07 15:45:23.850486 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:23.850401    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bf4bc488b-q22mp" event={"ID":"3538f866-cd99-4f24-ad38-fa444a4fdb05","Type":"ContainerStarted","Data":"a6049cd4ba02d95fc57928d396f5c8b2f3b3ac56cff9976f8a32c6e6ac0aa47d"}
Jul 07 15:45:23.850486 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:23.850420    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bf4bc488b-q22mp" event={"ID":"3538f866-cd99-4f24-ad38-fa444a4fdb05","Type":"ContainerStarted","Data":"6108e97618612ec0b5e7138a4d30f02831d53a9eaf56a5f0089a02dcc1c149f2"}
Jul 07 15:45:23.851969 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:23.851947    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6567dcbfb8-xb7hb" event={"ID":"106ff6c4-e1d4-4984-b495-65773809eff9","Type":"ContainerStarted","Data":"7f69d2e1063243be60aea45632fa425e17803175da960a781190c8a27b3f9bbf"}
Jul 07 15:45:23.853307 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:23.853285    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6bffdf9778-6qpwk" event={"ID":"d66b1fe3-70c4-422d-b909-9555280a9d1c","Type":"ContainerStarted","Data":"cee52576a7fe877955a9a1429bcebc2701281ea132c16971535b8e96b1e75d7b"}
Jul 07 15:45:23.854626 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:23.854605    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f9998b7cf-8w24j" event={"ID":"8c3fa5f2-200d-4b2c-adb8-5eb2a5ab9385","Type":"ContainerStarted","Data":"56b5f2908695f929fd73b392344c9ee135e4b41499584b7138292f0b313a6ba1"}
Jul 07 15:45:23.854733 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:23.854652    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f9998b7cf-8w24j"
Jul 07 15:45:23.856384 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:23.856366    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f9998b7cf-8w24j"
Jul 07 15:45:23.864547 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:23.864497    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-fvkrg" podStartSLOduration=66.550292101 podStartE2EDuration="1m16.864476868s" totalImagesPullingTime="10.314184767s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 15:44:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:45:23.864440203 +0000 UTC m=+77.952865874" watchObservedRunningTime="2026-07-07 15:45:23.864476868 +0000 UTC m=+77.952902539"
Jul 07 15:45:23.883021 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:23.882963    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f9998b7cf-8w24j" podStartSLOduration=2.016685653 podStartE2EDuration="10.882944488s" totalImagesPullingTime="8.866258835s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:45:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:45:23.882109683 +0000 UTC m=+77.970535353" watchObservedRunningTime="2026-07-07 15:45:23.882944488 +0000 UTC m=+77.971370198"
Jul 07 15:45:23.898789 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:23.898731    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6bffdf9778-6qpwk" podStartSLOduration=2.138511866 podStartE2EDuration="10.898710671s" totalImagesPullingTime="8.760198805s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:45:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:45:23.898463634 +0000 UTC m=+77.986889305" watchObservedRunningTime="2026-07-07 15:45:23.898710671 +0000 UTC m=+77.987136342"
Jul 07 15:45:23.915368 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:23.915313    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6567dcbfb8-xb7hb" podStartSLOduration=2.093070235 podStartE2EDuration="10.915292256s" totalImagesPullingTime="8.822222021s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:45:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:45:23.914464863 +0000 UTC m=+78.002890537" watchObservedRunningTime="2026-07-07 15:45:23.915292256 +0000 UTC m=+78.003717923"
Jul 07 15:45:23.936169 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:23.936117    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bf4bc488b-q22mp" podStartSLOduration=2.162968576 podStartE2EDuration="10.936096353s" totalImagesPullingTime="8.773127777s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:45:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:45:23.935094753 +0000 UTC m=+78.023520422" watchObservedRunningTime="2026-07-07 15:45:23.936096353 +0000 UTC m=+78.024522020"
Jul 07 15:45:23.955482 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:23.955406    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-tx5v9" podStartSLOduration=66.925147991 podStartE2EDuration="1m16.95538201s" totalImagesPullingTime="10.030234019s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:44:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:45:23.954574702 +0000 UTC m=+78.043000372" watchObservedRunningTime="2026-07-07 15:45:23.95538201 +0000 UTC m=+78.043807682"
Jul 07 15:45:27.399249 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:27.399217    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-t4tv9"]
Jul 07 15:45:27.402323 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:27.402306    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-t4tv9"
Jul 07 15:45:27.411546 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:27.411528    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 07 15:45:27.411951 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:27.411928    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 07 15:45:27.412057 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:27.411962    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 07 15:45:27.412383 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:27.412367    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 07 15:45:27.412475 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:27.412366    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 07 15:45:27.412561 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:27.412415    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 07 15:45:27.412659 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:27.412646    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-l97nq\""
Jul 07 15:45:27.502038 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:27.502007    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4c27c86b-6f34-4bb7-ae7b-1da713caf840-sys\") pod \"node-exporter-t4tv9\" (UID: \"4c27c86b-6f34-4bb7-ae7b-1da713caf840\") " pod="openshift-monitoring/node-exporter-t4tv9"
Jul 07 15:45:27.502038 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:27.502041    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4c27c86b-6f34-4bb7-ae7b-1da713caf840-node-exporter-wtmp\") pod \"node-exporter-t4tv9\" (UID: \"4c27c86b-6f34-4bb7-ae7b-1da713caf840\") " pod="openshift-monitoring/node-exporter-t4tv9"
Jul 07 15:45:27.502276 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:27.502060    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4c27c86b-6f34-4bb7-ae7b-1da713caf840-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-t4tv9\" (UID: \"4c27c86b-6f34-4bb7-ae7b-1da713caf840\") " pod="openshift-monitoring/node-exporter-t4tv9"
Jul 07 15:45:27.502276 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:27.502087    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4c27c86b-6f34-4bb7-ae7b-1da713caf840-node-exporter-textfile\") pod \"node-exporter-t4tv9\" (UID: \"4c27c86b-6f34-4bb7-ae7b-1da713caf840\") " pod="openshift-monitoring/node-exporter-t4tv9"
Jul 07 15:45:27.502276 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:27.502191    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nzpl\" (UniqueName: \"kubernetes.io/projected/4c27c86b-6f34-4bb7-ae7b-1da713caf840-kube-api-access-2nzpl\") pod \"node-exporter-t4tv9\" (UID: \"4c27c86b-6f34-4bb7-ae7b-1da713caf840\") " pod="openshift-monitoring/node-exporter-t4tv9"
Jul 07 15:45:27.502276 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:27.502245    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4c27c86b-6f34-4bb7-ae7b-1da713caf840-metrics-client-ca\") pod \"node-exporter-t4tv9\" (UID: \"4c27c86b-6f34-4bb7-ae7b-1da713caf840\") " pod="openshift-monitoring/node-exporter-t4tv9"
Jul 07 15:45:27.502458 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:27.502314    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4c27c86b-6f34-4bb7-ae7b-1da713caf840-node-exporter-accelerators-collector-config\") pod \"node-exporter-t4tv9\" (UID: \"4c27c86b-6f34-4bb7-ae7b-1da713caf840\") " pod="openshift-monitoring/node-exporter-t4tv9"
Jul 07 15:45:27.502458 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:27.502350    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4c27c86b-6f34-4bb7-ae7b-1da713caf840-root\") pod \"node-exporter-t4tv9\" (UID: \"4c27c86b-6f34-4bb7-ae7b-1da713caf840\") " pod="openshift-monitoring/node-exporter-t4tv9"
Jul 07 15:45:27.502458 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:27.502376    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4c27c86b-6f34-4bb7-ae7b-1da713caf840-node-exporter-tls\") pod \"node-exporter-t4tv9\" (UID: \"4c27c86b-6f34-4bb7-ae7b-1da713caf840\") " pod="openshift-monitoring/node-exporter-t4tv9"
Jul 07 15:45:27.603455 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:27.603411    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4c27c86b-6f34-4bb7-ae7b-1da713caf840-node-exporter-accelerators-collector-config\") pod \"node-exporter-t4tv9\" (UID: \"4c27c86b-6f34-4bb7-ae7b-1da713caf840\") " pod="openshift-monitoring/node-exporter-t4tv9"
Jul 07 15:45:27.603455 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:27.603454    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4c27c86b-6f34-4bb7-ae7b-1da713caf840-root\") pod \"node-exporter-t4tv9\" (UID: \"4c27c86b-6f34-4bb7-ae7b-1da713caf840\") " pod="openshift-monitoring/node-exporter-t4tv9"
Jul 07 15:45:27.603694 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:27.603474    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4c27c86b-6f34-4bb7-ae7b-1da713caf840-node-exporter-tls\") pod \"node-exporter-t4tv9\" (UID: \"4c27c86b-6f34-4bb7-ae7b-1da713caf840\") " pod="openshift-monitoring/node-exporter-t4tv9"
Jul 07 15:45:27.603694 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:27.603494    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4c27c86b-6f34-4bb7-ae7b-1da713caf840-sys\") pod \"node-exporter-t4tv9\" (UID: \"4c27c86b-6f34-4bb7-ae7b-1da713caf840\") " pod="openshift-monitoring/node-exporter-t4tv9"
Jul 07 15:45:27.603694 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:27.603515    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4c27c86b-6f34-4bb7-ae7b-1da713caf840-node-exporter-wtmp\") pod \"node-exporter-t4tv9\" (UID: \"4c27c86b-6f34-4bb7-ae7b-1da713caf840\") " pod="openshift-monitoring/node-exporter-t4tv9"
Jul 07 15:45:27.603694 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:27.603533    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4c27c86b-6f34-4bb7-ae7b-1da713caf840-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-t4tv9\" (UID: \"4c27c86b-6f34-4bb7-ae7b-1da713caf840\") " pod="openshift-monitoring/node-exporter-t4tv9"
Jul 07 15:45:27.603694 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:27.603548    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4c27c86b-6f34-4bb7-ae7b-1da713caf840-root\") pod \"node-exporter-t4tv9\" (UID: \"4c27c86b-6f34-4bb7-ae7b-1da713caf840\") " pod="openshift-monitoring/node-exporter-t4tv9"
Jul 07 15:45:27.603694 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:27.603560    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4c27c86b-6f34-4bb7-ae7b-1da713caf840-sys\") pod \"node-exporter-t4tv9\" (UID: \"4c27c86b-6f34-4bb7-ae7b-1da713caf840\") " pod="openshift-monitoring/node-exporter-t4tv9"
Jul 07 15:45:27.603694 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:27.603557    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4c27c86b-6f34-4bb7-ae7b-1da713caf840-node-exporter-textfile\") pod \"node-exporter-t4tv9\" (UID: \"4c27c86b-6f34-4bb7-ae7b-1da713caf840\") " pod="openshift-monitoring/node-exporter-t4tv9"
Jul 07 15:45:27.603694 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:45:27.603594    2566 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 07 15:45:27.603694 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:27.603633    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nzpl\" (UniqueName: \"kubernetes.io/projected/4c27c86b-6f34-4bb7-ae7b-1da713caf840-kube-api-access-2nzpl\") pod \"node-exporter-t4tv9\" (UID: \"4c27c86b-6f34-4bb7-ae7b-1da713caf840\") " pod="openshift-monitoring/node-exporter-t4tv9"
Jul 07 15:45:27.603694 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:45:27.603661    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c27c86b-6f34-4bb7-ae7b-1da713caf840-node-exporter-tls podName:4c27c86b-6f34-4bb7-ae7b-1da713caf840 nodeName:}" failed. No retries permitted until 2026-07-07 15:45:28.10363867 +0000 UTC m=+82.192064337 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/4c27c86b-6f34-4bb7-ae7b-1da713caf840-node-exporter-tls") pod "node-exporter-t4tv9" (UID: "4c27c86b-6f34-4bb7-ae7b-1da713caf840") : secret "node-exporter-tls" not found
Jul 07 15:45:27.604111 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:27.603708    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4c27c86b-6f34-4bb7-ae7b-1da713caf840-node-exporter-wtmp\") pod \"node-exporter-t4tv9\" (UID: \"4c27c86b-6f34-4bb7-ae7b-1da713caf840\") " pod="openshift-monitoring/node-exporter-t4tv9"
Jul 07 15:45:27.604111 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:27.603710    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4c27c86b-6f34-4bb7-ae7b-1da713caf840-metrics-client-ca\") pod \"node-exporter-t4tv9\" (UID: \"4c27c86b-6f34-4bb7-ae7b-1da713caf840\") " pod="openshift-monitoring/node-exporter-t4tv9"
Jul 07 15:45:27.604111 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:27.603917    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4c27c86b-6f34-4bb7-ae7b-1da713caf840-node-exporter-textfile\") pod \"node-exporter-t4tv9\" (UID: \"4c27c86b-6f34-4bb7-ae7b-1da713caf840\") " pod="openshift-monitoring/node-exporter-t4tv9"
Jul 07 15:45:27.604111 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:27.604076    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4c27c86b-6f34-4bb7-ae7b-1da713caf840-node-exporter-accelerators-collector-config\") pod \"node-exporter-t4tv9\" (UID: \"4c27c86b-6f34-4bb7-ae7b-1da713caf840\") " pod="openshift-monitoring/node-exporter-t4tv9"
Jul 07 15:45:27.604228 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:27.604188    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4c27c86b-6f34-4bb7-ae7b-1da713caf840-metrics-client-ca\") pod \"node-exporter-t4tv9\" (UID: \"4c27c86b-6f34-4bb7-ae7b-1da713caf840\") " pod="openshift-monitoring/node-exporter-t4tv9"
Jul 07 15:45:27.605927 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:27.605908    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4c27c86b-6f34-4bb7-ae7b-1da713caf840-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-t4tv9\" (UID: \"4c27c86b-6f34-4bb7-ae7b-1da713caf840\") " pod="openshift-monitoring/node-exporter-t4tv9"
Jul 07 15:45:27.620560 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:27.620540    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nzpl\" (UniqueName: \"kubernetes.io/projected/4c27c86b-6f34-4bb7-ae7b-1da713caf840-kube-api-access-2nzpl\") pod \"node-exporter-t4tv9\" (UID: \"4c27c86b-6f34-4bb7-ae7b-1da713caf840\") " pod="openshift-monitoring/node-exporter-t4tv9"
Jul 07 15:45:28.108910 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:28.108857    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4c27c86b-6f34-4bb7-ae7b-1da713caf840-node-exporter-tls\") pod \"node-exporter-t4tv9\" (UID: \"4c27c86b-6f34-4bb7-ae7b-1da713caf840\") " pod="openshift-monitoring/node-exporter-t4tv9"
Jul 07 15:45:28.111284 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:28.111258    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4c27c86b-6f34-4bb7-ae7b-1da713caf840-node-exporter-tls\") pod \"node-exporter-t4tv9\" (UID: \"4c27c86b-6f34-4bb7-ae7b-1da713caf840\") " pod="openshift-monitoring/node-exporter-t4tv9"
Jul 07 15:45:28.311309 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:28.311181    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-t4tv9"
Jul 07 15:45:28.322919 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:45:28.322876    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c27c86b_6f34_4bb7_ae7b_1da713caf840.slice/crio-2c8ef5c38c73503196a28f85a8aa2a996a5f92703a3d8807821f5d624ab15825 WatchSource:0}: Error finding container 2c8ef5c38c73503196a28f85a8aa2a996a5f92703a3d8807821f5d624ab15825: Status 404 returned error can't find the container with id 2c8ef5c38c73503196a28f85a8aa2a996a5f92703a3d8807821f5d624ab15825
Jul 07 15:45:28.875540 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:28.875503    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-t4tv9" event={"ID":"4c27c86b-6f34-4bb7-ae7b-1da713caf840","Type":"ContainerStarted","Data":"2c8ef5c38c73503196a28f85a8aa2a996a5f92703a3d8807821f5d624ab15825"}
Jul 07 15:45:29.882519 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:29.882438    2566 generic.go:356] "Generic (PLEG): container finished" podID="4c27c86b-6f34-4bb7-ae7b-1da713caf840" containerID="5ad6d06e3b4c72b440f66458e995a78f97410becefd400f6914036188ee5880f" exitCode=0
Jul 07 15:45:29.882519 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:29.882475    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-t4tv9" event={"ID":"4c27c86b-6f34-4bb7-ae7b-1da713caf840","Type":"ContainerDied","Data":"5ad6d06e3b4c72b440f66458e995a78f97410becefd400f6914036188ee5880f"}
Jul 07 15:45:30.888660 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:30.888622    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-t4tv9" event={"ID":"4c27c86b-6f34-4bb7-ae7b-1da713caf840","Type":"ContainerStarted","Data":"234159a6deab000f63e1a44a303e440b31998c6c56199d059bb5316fe0d29dce"}
Jul 07 15:45:30.889069 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:30.888674    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-t4tv9" event={"ID":"4c27c86b-6f34-4bb7-ae7b-1da713caf840","Type":"ContainerStarted","Data":"cec28fd0a656120b255525ffe9fcb5b7632bd36c1f2c5a231a59c8feb1f0118b"}
Jul 07 15:45:30.910024 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:30.909976    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-t4tv9" podStartSLOduration=2.586168132 podStartE2EDuration="3.909960606s" totalImagesPullingTime="1.260357317s" totalInitContainerRuntime="63.435157ms" isStatefulPod=true podCreationTimestamp="2026-07-07 15:45:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:45:30.90922567 +0000 UTC m=+84.997651379" watchObservedRunningTime="2026-07-07 15:45:30.909960606 +0000 UTC m=+84.998386289"
Jul 07 15:45:36.823445 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:36.823414    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-77d7b5bb7d-rcmxv"
Jul 07 15:45:38.433057 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:38.432993    2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7c68df6494-4mrwk" podUID="54bc412f-9e52-45a2-b820-3dee0d9902d7" containerName="registry" containerID="cri-o://f2f62de2796a7d39c708647630c9c3465ea65d3682c5a4be91a347a7f4046135" gracePeriod=30
Jul 07 15:45:38.694702 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:38.694644    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c68df6494-4mrwk"
Jul 07 15:45:38.798775 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:38.798745    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/54bc412f-9e52-45a2-b820-3dee0d9902d7-registry-tls\") pod \"54bc412f-9e52-45a2-b820-3dee0d9902d7\" (UID: \"54bc412f-9e52-45a2-b820-3dee0d9902d7\") "
Jul 07 15:45:38.798962 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:38.798803    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/54bc412f-9e52-45a2-b820-3dee0d9902d7-trusted-ca\") pod \"54bc412f-9e52-45a2-b820-3dee0d9902d7\" (UID: \"54bc412f-9e52-45a2-b820-3dee0d9902d7\") "
Jul 07 15:45:38.798962 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:38.798831    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/54bc412f-9e52-45a2-b820-3dee0d9902d7-image-registry-private-configuration\") pod \"54bc412f-9e52-45a2-b820-3dee0d9902d7\" (UID: \"54bc412f-9e52-45a2-b820-3dee0d9902d7\") "
Jul 07 15:45:38.798962 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:38.798872    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bl89k\" (UniqueName: \"kubernetes.io/projected/54bc412f-9e52-45a2-b820-3dee0d9902d7-kube-api-access-bl89k\") pod \"54bc412f-9e52-45a2-b820-3dee0d9902d7\" (UID: \"54bc412f-9e52-45a2-b820-3dee0d9902d7\") "
Jul 07 15:45:38.798962 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:38.798922    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/54bc412f-9e52-45a2-b820-3dee0d9902d7-registry-certificates\") pod \"54bc412f-9e52-45a2-b820-3dee0d9902d7\" (UID: \"54bc412f-9e52-45a2-b820-3dee0d9902d7\") "
Jul 07 15:45:38.799236 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:38.799045    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/54bc412f-9e52-45a2-b820-3dee0d9902d7-ca-trust-extracted\") pod \"54bc412f-9e52-45a2-b820-3dee0d9902d7\" (UID: \"54bc412f-9e52-45a2-b820-3dee0d9902d7\") "
Jul 07 15:45:38.799236 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:38.799093    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/54bc412f-9e52-45a2-b820-3dee0d9902d7-installation-pull-secrets\") pod \"54bc412f-9e52-45a2-b820-3dee0d9902d7\" (UID: \"54bc412f-9e52-45a2-b820-3dee0d9902d7\") "
Jul 07 15:45:38.799236 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:38.799118    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/54bc412f-9e52-45a2-b820-3dee0d9902d7-bound-sa-token\") pod \"54bc412f-9e52-45a2-b820-3dee0d9902d7\" (UID: \"54bc412f-9e52-45a2-b820-3dee0d9902d7\") "
Jul 07 15:45:38.799395 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:38.799331    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54bc412f-9e52-45a2-b820-3dee0d9902d7-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "54bc412f-9e52-45a2-b820-3dee0d9902d7" (UID: "54bc412f-9e52-45a2-b820-3dee0d9902d7"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 15:45:38.799633 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:38.799608    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54bc412f-9e52-45a2-b820-3dee0d9902d7-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "54bc412f-9e52-45a2-b820-3dee0d9902d7" (UID: "54bc412f-9e52-45a2-b820-3dee0d9902d7"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 15:45:38.801459 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:38.801420    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54bc412f-9e52-45a2-b820-3dee0d9902d7-kube-api-access-bl89k" (OuterVolumeSpecName: "kube-api-access-bl89k") pod "54bc412f-9e52-45a2-b820-3dee0d9902d7" (UID: "54bc412f-9e52-45a2-b820-3dee0d9902d7"). InnerVolumeSpecName "kube-api-access-bl89k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 15:45:38.801604 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:38.801579    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54bc412f-9e52-45a2-b820-3dee0d9902d7-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "54bc412f-9e52-45a2-b820-3dee0d9902d7" (UID: "54bc412f-9e52-45a2-b820-3dee0d9902d7"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 15:45:38.801648 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:38.801584    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54bc412f-9e52-45a2-b820-3dee0d9902d7-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "54bc412f-9e52-45a2-b820-3dee0d9902d7" (UID: "54bc412f-9e52-45a2-b820-3dee0d9902d7"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 15:45:38.801764 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:38.801746    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54bc412f-9e52-45a2-b820-3dee0d9902d7-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "54bc412f-9e52-45a2-b820-3dee0d9902d7" (UID: "54bc412f-9e52-45a2-b820-3dee0d9902d7"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 15:45:38.801855 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:38.801830    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54bc412f-9e52-45a2-b820-3dee0d9902d7-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "54bc412f-9e52-45a2-b820-3dee0d9902d7" (UID: "54bc412f-9e52-45a2-b820-3dee0d9902d7"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 15:45:38.811036 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:38.811008    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54bc412f-9e52-45a2-b820-3dee0d9902d7-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "54bc412f-9e52-45a2-b820-3dee0d9902d7" (UID: "54bc412f-9e52-45a2-b820-3dee0d9902d7"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 15:45:38.900069 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:38.900036    2566 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/54bc412f-9e52-45a2-b820-3dee0d9902d7-registry-tls\") on node \"ip-10-0-138-101.ec2.internal\" DevicePath \"\""
Jul 07 15:45:38.900069 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:38.900060    2566 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/54bc412f-9e52-45a2-b820-3dee0d9902d7-trusted-ca\") on node \"ip-10-0-138-101.ec2.internal\" DevicePath \"\""
Jul 07 15:45:38.900069 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:38.900070    2566 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/54bc412f-9e52-45a2-b820-3dee0d9902d7-image-registry-private-configuration\") on node \"ip-10-0-138-101.ec2.internal\" DevicePath \"\""
Jul 07 15:45:38.900069 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:38.900079    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bl89k\" (UniqueName: \"kubernetes.io/projected/54bc412f-9e52-45a2-b820-3dee0d9902d7-kube-api-access-bl89k\") on node \"ip-10-0-138-101.ec2.internal\" DevicePath \"\""
Jul 07 15:45:38.900310 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:38.900089    2566 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/54bc412f-9e52-45a2-b820-3dee0d9902d7-registry-certificates\") on node \"ip-10-0-138-101.ec2.internal\" DevicePath \"\""
Jul 07 15:45:38.900310 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:38.900099    2566 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/54bc412f-9e52-45a2-b820-3dee0d9902d7-ca-trust-extracted\") on node \"ip-10-0-138-101.ec2.internal\" DevicePath \"\""
Jul 07 15:45:38.900310 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:38.900107    2566 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/54bc412f-9e52-45a2-b820-3dee0d9902d7-installation-pull-secrets\") on node \"ip-10-0-138-101.ec2.internal\" DevicePath \"\""
Jul 07 15:45:38.900310 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:38.900117    2566 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/54bc412f-9e52-45a2-b820-3dee0d9902d7-bound-sa-token\") on node \"ip-10-0-138-101.ec2.internal\" DevicePath \"\""
Jul 07 15:45:38.919300 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:38.919191    2566 generic.go:356] "Generic (PLEG): container finished" podID="54bc412f-9e52-45a2-b820-3dee0d9902d7" containerID="f2f62de2796a7d39c708647630c9c3465ea65d3682c5a4be91a347a7f4046135" exitCode=0
Jul 07 15:45:38.919614 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:38.919559    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c68df6494-4mrwk" event={"ID":"54bc412f-9e52-45a2-b820-3dee0d9902d7","Type":"ContainerDied","Data":"f2f62de2796a7d39c708647630c9c3465ea65d3682c5a4be91a347a7f4046135"}
Jul 07 15:45:38.919848 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:38.919747    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c68df6494-4mrwk" event={"ID":"54bc412f-9e52-45a2-b820-3dee0d9902d7","Type":"ContainerDied","Data":"1e9a99dac28b13beb4efd850bc960621fc758b5db40076b0b9ac2d7c97326b9e"}
Jul 07 15:45:38.919848 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:38.919804    2566 scope.go:117] "RemoveContainer" containerID="f2f62de2796a7d39c708647630c9c3465ea65d3682c5a4be91a347a7f4046135"
Jul 07 15:45:38.920263 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:38.920212    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c68df6494-4mrwk"
Jul 07 15:45:38.932108 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:38.932088    2566 scope.go:117] "RemoveContainer" containerID="f2f62de2796a7d39c708647630c9c3465ea65d3682c5a4be91a347a7f4046135"
Jul 07 15:45:38.934723 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:45:38.934697    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2f62de2796a7d39c708647630c9c3465ea65d3682c5a4be91a347a7f4046135\": container with ID starting with f2f62de2796a7d39c708647630c9c3465ea65d3682c5a4be91a347a7f4046135 not found: ID does not exist" containerID="f2f62de2796a7d39c708647630c9c3465ea65d3682c5a4be91a347a7f4046135"
Jul 07 15:45:38.934812 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:38.934735    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2f62de2796a7d39c708647630c9c3465ea65d3682c5a4be91a347a7f4046135"} err="failed to get container status \"f2f62de2796a7d39c708647630c9c3465ea65d3682c5a4be91a347a7f4046135\": rpc error: code = NotFound desc = could not find container \"f2f62de2796a7d39c708647630c9c3465ea65d3682c5a4be91a347a7f4046135\": container with ID starting with f2f62de2796a7d39c708647630c9c3465ea65d3682c5a4be91a347a7f4046135 not found: ID does not exist"
Jul 07 15:45:38.944165 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:38.944142    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7c68df6494-4mrwk"]
Jul 07 15:45:38.946564 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:38.946505    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7c68df6494-4mrwk"]
Jul 07 15:45:40.472702 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:40.472662    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54bc412f-9e52-45a2-b820-3dee0d9902d7" path="/var/lib/kubelet/pods/54bc412f-9e52-45a2-b820-3dee0d9902d7/volumes"
Jul 07 15:45:54.861859 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:45:54.861826    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-tx5v9"
Jul 07 15:49:06.337059 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:49:06.337032    2566 kubelet.go:1648] "Image garbage collection succeeded"
Jul 07 15:49:41.231408 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:49:41.231375    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-56f57b78d4-tbrcq"]
Jul 07 15:49:41.232666 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:49:41.231596    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54bc412f-9e52-45a2-b820-3dee0d9902d7" containerName="registry"
Jul 07 15:49:41.232666 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:49:41.231606    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="54bc412f-9e52-45a2-b820-3dee0d9902d7" containerName="registry"
Jul 07 15:49:41.232666 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:49:41.231656    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="54bc412f-9e52-45a2-b820-3dee0d9902d7" containerName="registry"
Jul 07 15:49:41.233571 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:49:41.233551    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-56f57b78d4-tbrcq"
Jul 07 15:49:41.236155 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:49:41.236094    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 07 15:49:41.236286 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:49:41.236155    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-hzz2w\""
Jul 07 15:49:41.236375 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:49:41.236359    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 07 15:49:41.236455 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:49:41.236369    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 07 15:49:41.236455 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:49:41.236423    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 07 15:49:41.247742 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:49:41.247721    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-56f57b78d4-tbrcq"]
Jul 07 15:49:41.309489 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:49:41.309456    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7b3312b8-7ba2-4102-b36f-3b0f58178f4e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-56f57b78d4-tbrcq\" (UID: \"7b3312b8-7ba2-4102-b36f-3b0f58178f4e\") " pod="opendatahub/opendatahub-operator-controller-manager-56f57b78d4-tbrcq"
Jul 07 15:49:41.309660 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:49:41.309498    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7b3312b8-7ba2-4102-b36f-3b0f58178f4e-webhook-cert\") pod \"opendatahub-operator-controller-manager-56f57b78d4-tbrcq\" (UID: \"7b3312b8-7ba2-4102-b36f-3b0f58178f4e\") " pod="opendatahub/opendatahub-operator-controller-manager-56f57b78d4-tbrcq"
Jul 07 15:49:41.309660 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:49:41.309524    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49gr5\" (UniqueName: \"kubernetes.io/projected/7b3312b8-7ba2-4102-b36f-3b0f58178f4e-kube-api-access-49gr5\") pod \"opendatahub-operator-controller-manager-56f57b78d4-tbrcq\" (UID: \"7b3312b8-7ba2-4102-b36f-3b0f58178f4e\") " pod="opendatahub/opendatahub-operator-controller-manager-56f57b78d4-tbrcq"
Jul 07 15:49:41.309660 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:49:41.309613    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7b3312b8-7ba2-4102-b36f-3b0f58178f4e-manifests\") pod \"opendatahub-operator-controller-manager-56f57b78d4-tbrcq\" (UID: \"7b3312b8-7ba2-4102-b36f-3b0f58178f4e\") " pod="opendatahub/opendatahub-operator-controller-manager-56f57b78d4-tbrcq"
Jul 07 15:49:41.309660 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:49:41.309644    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7b3312b8-7ba2-4102-b36f-3b0f58178f4e-tmp\") pod \"opendatahub-operator-controller-manager-56f57b78d4-tbrcq\" (UID: \"7b3312b8-7ba2-4102-b36f-3b0f58178f4e\") " pod="opendatahub/opendatahub-operator-controller-manager-56f57b78d4-tbrcq"
Jul 07 15:49:41.410845 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:49:41.410813    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7b3312b8-7ba2-4102-b36f-3b0f58178f4e-webhook-cert\") pod \"opendatahub-operator-controller-manager-56f57b78d4-tbrcq\" (UID: \"7b3312b8-7ba2-4102-b36f-3b0f58178f4e\") " pod="opendatahub/opendatahub-operator-controller-manager-56f57b78d4-tbrcq"
Jul 07 15:49:41.410845 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:49:41.410848    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49gr5\" (UniqueName: \"kubernetes.io/projected/7b3312b8-7ba2-4102-b36f-3b0f58178f4e-kube-api-access-49gr5\") pod \"opendatahub-operator-controller-manager-56f57b78d4-tbrcq\" (UID: \"7b3312b8-7ba2-4102-b36f-3b0f58178f4e\") " pod="opendatahub/opendatahub-operator-controller-manager-56f57b78d4-tbrcq"
Jul 07 15:49:41.411079 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:49:41.410880    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7b3312b8-7ba2-4102-b36f-3b0f58178f4e-manifests\") pod \"opendatahub-operator-controller-manager-56f57b78d4-tbrcq\" (UID: \"7b3312b8-7ba2-4102-b36f-3b0f58178f4e\") " pod="opendatahub/opendatahub-operator-controller-manager-56f57b78d4-tbrcq"
Jul 07 15:49:41.411079 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:49:41.410925    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7b3312b8-7ba2-4102-b36f-3b0f58178f4e-tmp\") pod \"opendatahub-operator-controller-manager-56f57b78d4-tbrcq\" (UID: \"7b3312b8-7ba2-4102-b36f-3b0f58178f4e\") " pod="opendatahub/opendatahub-operator-controller-manager-56f57b78d4-tbrcq"
Jul 07 15:49:41.411079 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:49:41.410969    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7b3312b8-7ba2-4102-b36f-3b0f58178f4e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-56f57b78d4-tbrcq\" (UID: \"7b3312b8-7ba2-4102-b36f-3b0f58178f4e\") " pod="opendatahub/opendatahub-operator-controller-manager-56f57b78d4-tbrcq"
Jul 07 15:49:41.411450 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:49:41.411420    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7b3312b8-7ba2-4102-b36f-3b0f58178f4e-manifests\") pod \"opendatahub-operator-controller-manager-56f57b78d4-tbrcq\" (UID: \"7b3312b8-7ba2-4102-b36f-3b0f58178f4e\") " pod="opendatahub/opendatahub-operator-controller-manager-56f57b78d4-tbrcq"
Jul 07 15:49:41.413259 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:49:41.413230    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7b3312b8-7ba2-4102-b36f-3b0f58178f4e-tmp\") pod \"opendatahub-operator-controller-manager-56f57b78d4-tbrcq\" (UID: \"7b3312b8-7ba2-4102-b36f-3b0f58178f4e\") " pod="opendatahub/opendatahub-operator-controller-manager-56f57b78d4-tbrcq"
Jul 07 15:49:41.413352 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:49:41.413262    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7b3312b8-7ba2-4102-b36f-3b0f58178f4e-webhook-cert\") pod \"opendatahub-operator-controller-manager-56f57b78d4-tbrcq\" (UID: \"7b3312b8-7ba2-4102-b36f-3b0f58178f4e\") " pod="opendatahub/opendatahub-operator-controller-manager-56f57b78d4-tbrcq"
Jul 07 15:49:41.413352 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:49:41.413345    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7b3312b8-7ba2-4102-b36f-3b0f58178f4e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-56f57b78d4-tbrcq\" (UID: \"7b3312b8-7ba2-4102-b36f-3b0f58178f4e\") " pod="opendatahub/opendatahub-operator-controller-manager-56f57b78d4-tbrcq"
Jul 07 15:49:41.418793 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:49:41.418775    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49gr5\" (UniqueName: \"kubernetes.io/projected/7b3312b8-7ba2-4102-b36f-3b0f58178f4e-kube-api-access-49gr5\") pod \"opendatahub-operator-controller-manager-56f57b78d4-tbrcq\" (UID: \"7b3312b8-7ba2-4102-b36f-3b0f58178f4e\") " pod="opendatahub/opendatahub-operator-controller-manager-56f57b78d4-tbrcq"
Jul 07 15:49:41.543198 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:49:41.543121    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-56f57b78d4-tbrcq"
Jul 07 15:49:41.679140 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:49:41.679004    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-56f57b78d4-tbrcq"]
Jul 07 15:49:41.682184 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:49:41.682158    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b3312b8_7ba2_4102_b36f_3b0f58178f4e.slice/crio-4dfd384c9e14613d329b8a5cad3cec3fe57c93b1ab9dbe9ff3d127c91386a214 WatchSource:0}: Error finding container 4dfd384c9e14613d329b8a5cad3cec3fe57c93b1ab9dbe9ff3d127c91386a214: Status 404 returned error can't find the container with id 4dfd384c9e14613d329b8a5cad3cec3fe57c93b1ab9dbe9ff3d127c91386a214
Jul 07 15:49:41.683930 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:49:41.683913    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 15:49:41.710565 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:49:41.710536    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-56f57b78d4-tbrcq" event={"ID":"7b3312b8-7ba2-4102-b36f-3b0f58178f4e","Type":"ContainerStarted","Data":"4dfd384c9e14613d329b8a5cad3cec3fe57c93b1ab9dbe9ff3d127c91386a214"}
Jul 07 15:49:44.724014 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:49:44.723975    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-56f57b78d4-tbrcq" event={"ID":"7b3312b8-7ba2-4102-b36f-3b0f58178f4e","Type":"ContainerStarted","Data":"708b88458a91c4ecaf18a8ecfa70c43263bd2a893b9051842908c03a754c4970"}
Jul 07 15:49:45.729167 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:49:45.729138    2566 generic.go:356] "Generic (PLEG): container finished" podID="7b3312b8-7ba2-4102-b36f-3b0f58178f4e" containerID="708b88458a91c4ecaf18a8ecfa70c43263bd2a893b9051842908c03a754c4970" exitCode=0
Jul 07 15:49:45.729566 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:49:45.729181    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-56f57b78d4-tbrcq" event={"ID":"7b3312b8-7ba2-4102-b36f-3b0f58178f4e","Type":"ContainerDied","Data":"708b88458a91c4ecaf18a8ecfa70c43263bd2a893b9051842908c03a754c4970"}
Jul 07 15:49:46.734063 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:49:46.734031    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-56f57b78d4-tbrcq" event={"ID":"7b3312b8-7ba2-4102-b36f-3b0f58178f4e","Type":"ContainerStarted","Data":"80432d05f528e54f54b5a8e4aeb5449a6c99231a6b29087dc03d712e6eb759f8"}
Jul 07 15:49:46.734421 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:49:46.734162    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-56f57b78d4-tbrcq"
Jul 07 15:49:46.756212 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:49:46.756168    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-56f57b78d4-tbrcq" podStartSLOduration=1.293914701 podStartE2EDuration="5.756153169s" totalImagesPullingTime="3.386619253s" totalInitContainerRuntime="1.075619215s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:49:41 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:49:46.755504739 +0000 UTC m=+340.843930411" watchObservedRunningTime="2026-07-07 15:49:46.756153169 +0000 UTC m=+340.844578838"
Jul 07 15:49:57.739332 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:49:57.739303    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-56f57b78d4-tbrcq"
Jul 07 15:50:00.812399 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:00.812356    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-59ffcfbcb9-n7k4p"]
Jul 07 15:50:00.815050 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:00.815030    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-59ffcfbcb9-n7k4p"
Jul 07 15:50:00.818954 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:00.818930    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 07 15:50:00.819093 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:00.818976    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 07 15:50:00.821280 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:00.821254    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 07 15:50:00.825406 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:00.825388    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 07 15:50:00.837049 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:00.837026    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-59ffcfbcb9-n7k4p"]
Jul 07 15:50:00.853978 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:00.853953    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18a0d440-92e1-41b8-91ab-b9bb82c6d229-tls-certs\") pod \"kube-auth-proxy-59ffcfbcb9-n7k4p\" (UID: \"18a0d440-92e1-41b8-91ab-b9bb82c6d229\") " pod="openshift-ingress/kube-auth-proxy-59ffcfbcb9-n7k4p"
Jul 07 15:50:00.854144 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:00.853983    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2mf2\" (UniqueName: \"kubernetes.io/projected/18a0d440-92e1-41b8-91ab-b9bb82c6d229-kube-api-access-d2mf2\") pod \"kube-auth-proxy-59ffcfbcb9-n7k4p\" (UID: \"18a0d440-92e1-41b8-91ab-b9bb82c6d229\") " pod="openshift-ingress/kube-auth-proxy-59ffcfbcb9-n7k4p"
Jul 07 15:50:00.854144 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:00.854007    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/18a0d440-92e1-41b8-91ab-b9bb82c6d229-tmp\") pod \"kube-auth-proxy-59ffcfbcb9-n7k4p\" (UID: \"18a0d440-92e1-41b8-91ab-b9bb82c6d229\") " pod="openshift-ingress/kube-auth-proxy-59ffcfbcb9-n7k4p"
Jul 07 15:50:00.954944 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:00.954878    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18a0d440-92e1-41b8-91ab-b9bb82c6d229-tls-certs\") pod \"kube-auth-proxy-59ffcfbcb9-n7k4p\" (UID: \"18a0d440-92e1-41b8-91ab-b9bb82c6d229\") " pod="openshift-ingress/kube-auth-proxy-59ffcfbcb9-n7k4p"
Jul 07 15:50:00.954944 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:00.954945    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2mf2\" (UniqueName: \"kubernetes.io/projected/18a0d440-92e1-41b8-91ab-b9bb82c6d229-kube-api-access-d2mf2\") pod \"kube-auth-proxy-59ffcfbcb9-n7k4p\" (UID: \"18a0d440-92e1-41b8-91ab-b9bb82c6d229\") " pod="openshift-ingress/kube-auth-proxy-59ffcfbcb9-n7k4p"
Jul 07 15:50:00.955268 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:00.955111    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/18a0d440-92e1-41b8-91ab-b9bb82c6d229-tmp\") pod \"kube-auth-proxy-59ffcfbcb9-n7k4p\" (UID: \"18a0d440-92e1-41b8-91ab-b9bb82c6d229\") " pod="openshift-ingress/kube-auth-proxy-59ffcfbcb9-n7k4p"
Jul 07 15:50:00.957259 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:00.957230    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/18a0d440-92e1-41b8-91ab-b9bb82c6d229-tmp\") pod \"kube-auth-proxy-59ffcfbcb9-n7k4p\" (UID: \"18a0d440-92e1-41b8-91ab-b9bb82c6d229\") " pod="openshift-ingress/kube-auth-proxy-59ffcfbcb9-n7k4p"
Jul 07 15:50:00.957417 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:00.957397    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18a0d440-92e1-41b8-91ab-b9bb82c6d229-tls-certs\") pod \"kube-auth-proxy-59ffcfbcb9-n7k4p\" (UID: \"18a0d440-92e1-41b8-91ab-b9bb82c6d229\") " pod="openshift-ingress/kube-auth-proxy-59ffcfbcb9-n7k4p"
Jul 07 15:50:00.963582 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:00.963561    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2mf2\" (UniqueName: \"kubernetes.io/projected/18a0d440-92e1-41b8-91ab-b9bb82c6d229-kube-api-access-d2mf2\") pod \"kube-auth-proxy-59ffcfbcb9-n7k4p\" (UID: \"18a0d440-92e1-41b8-91ab-b9bb82c6d229\") " pod="openshift-ingress/kube-auth-proxy-59ffcfbcb9-n7k4p"
Jul 07 15:50:01.125747 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:01.125660    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-59ffcfbcb9-n7k4p"
Jul 07 15:50:01.175193 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:01.175138    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-kt87x"]
Jul 07 15:50:01.179493 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:01.179470    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-kt87x"
Jul 07 15:50:01.183109 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:01.181670    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Jul 07 15:50:01.183109 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:01.181853    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-758k2\""
Jul 07 15:50:01.189988 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:01.189733    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-kt87x"]
Jul 07 15:50:01.258149 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:01.258120    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2t5rs\" (UniqueName: \"kubernetes.io/projected/d1ce20f8-4500-4af3-b2e9-d5c6933df650-kube-api-access-2t5rs\") pod \"odh-model-controller-7998c67dc4-kt87x\" (UID: \"d1ce20f8-4500-4af3-b2e9-d5c6933df650\") " pod="opendatahub/odh-model-controller-7998c67dc4-kt87x"
Jul 07 15:50:01.258300 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:01.258198    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d1ce20f8-4500-4af3-b2e9-d5c6933df650-cert\") pod \"odh-model-controller-7998c67dc4-kt87x\" (UID: \"d1ce20f8-4500-4af3-b2e9-d5c6933df650\") " pod="opendatahub/odh-model-controller-7998c67dc4-kt87x"
Jul 07 15:50:01.260980 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:01.260958    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-59ffcfbcb9-n7k4p"]
Jul 07 15:50:01.263308 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:50:01.263283    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18a0d440_92e1_41b8_91ab_b9bb82c6d229.slice/crio-fd51dba3aa031f6ec1d874eef7800a5a361a09b6b9d54a93fdad6ca6878181be WatchSource:0}: Error finding container fd51dba3aa031f6ec1d874eef7800a5a361a09b6b9d54a93fdad6ca6878181be: Status 404 returned error can't find the container with id fd51dba3aa031f6ec1d874eef7800a5a361a09b6b9d54a93fdad6ca6878181be
Jul 07 15:50:01.358540 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:01.358504    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d1ce20f8-4500-4af3-b2e9-d5c6933df650-cert\") pod \"odh-model-controller-7998c67dc4-kt87x\" (UID: \"d1ce20f8-4500-4af3-b2e9-d5c6933df650\") " pod="opendatahub/odh-model-controller-7998c67dc4-kt87x"
Jul 07 15:50:01.358725 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:01.358559    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2t5rs\" (UniqueName: \"kubernetes.io/projected/d1ce20f8-4500-4af3-b2e9-d5c6933df650-kube-api-access-2t5rs\") pod \"odh-model-controller-7998c67dc4-kt87x\" (UID: \"d1ce20f8-4500-4af3-b2e9-d5c6933df650\") " pod="opendatahub/odh-model-controller-7998c67dc4-kt87x"
Jul 07 15:50:01.358725 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:50:01.358664    2566 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jul 07 15:50:01.358805 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:50:01.358729    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1ce20f8-4500-4af3-b2e9-d5c6933df650-cert podName:d1ce20f8-4500-4af3-b2e9-d5c6933df650 nodeName:}" failed. No retries permitted until 2026-07-07 15:50:01.858711711 +0000 UTC m=+355.947137359 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/d1ce20f8-4500-4af3-b2e9-d5c6933df650-cert") pod "odh-model-controller-7998c67dc4-kt87x" (UID: "d1ce20f8-4500-4af3-b2e9-d5c6933df650") : secret "odh-model-controller-webhook-cert" not found
Jul 07 15:50:01.367578 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:01.367548    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2t5rs\" (UniqueName: \"kubernetes.io/projected/d1ce20f8-4500-4af3-b2e9-d5c6933df650-kube-api-access-2t5rs\") pod \"odh-model-controller-7998c67dc4-kt87x\" (UID: \"d1ce20f8-4500-4af3-b2e9-d5c6933df650\") " pod="opendatahub/odh-model-controller-7998c67dc4-kt87x"
Jul 07 15:50:01.784135 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:01.784094    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-59ffcfbcb9-n7k4p" event={"ID":"18a0d440-92e1-41b8-91ab-b9bb82c6d229","Type":"ContainerStarted","Data":"fd51dba3aa031f6ec1d874eef7800a5a361a09b6b9d54a93fdad6ca6878181be"}
Jul 07 15:50:01.863785 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:01.863751    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d1ce20f8-4500-4af3-b2e9-d5c6933df650-cert\") pod \"odh-model-controller-7998c67dc4-kt87x\" (UID: \"d1ce20f8-4500-4af3-b2e9-d5c6933df650\") " pod="opendatahub/odh-model-controller-7998c67dc4-kt87x"
Jul 07 15:50:01.866115 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:01.866085    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d1ce20f8-4500-4af3-b2e9-d5c6933df650-cert\") pod \"odh-model-controller-7998c67dc4-kt87x\" (UID: \"d1ce20f8-4500-4af3-b2e9-d5c6933df650\") " pod="opendatahub/odh-model-controller-7998c67dc4-kt87x"
Jul 07 15:50:02.063376 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:02.063298    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-nkdqs"]
Jul 07 15:50:02.066412 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:02.066390    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-nkdqs"
Jul 07 15:50:02.069219 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:02.069193    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Jul 07 15:50:02.069393 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:02.069375    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-6mzh2\""
Jul 07 15:50:02.082328 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:02.082300    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-nkdqs"]
Jul 07 15:50:02.097585 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:02.097552    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-kt87x"
Jul 07 15:50:02.166081 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:02.166038    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5962e3e-69f6-42f4-b642-7bac15a37ede-tls-certs\") pod \"model-serving-api-54d89bd5fc-nkdqs\" (UID: \"e5962e3e-69f6-42f4-b642-7bac15a37ede\") " pod="opendatahub/model-serving-api-54d89bd5fc-nkdqs"
Jul 07 15:50:02.166249 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:02.166083    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsw9d\" (UniqueName: \"kubernetes.io/projected/e5962e3e-69f6-42f4-b642-7bac15a37ede-kube-api-access-nsw9d\") pod \"model-serving-api-54d89bd5fc-nkdqs\" (UID: \"e5962e3e-69f6-42f4-b642-7bac15a37ede\") " pod="opendatahub/model-serving-api-54d89bd5fc-nkdqs"
Jul 07 15:50:02.244131 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:02.244099    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-kt87x"]
Jul 07 15:50:02.247139 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:50:02.247114    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1ce20f8_4500_4af3_b2e9_d5c6933df650.slice/crio-db27187ce40855919b2198eca9fc2679666fe4ae1a8157a177bc6c53d3ff8425 WatchSource:0}: Error finding container db27187ce40855919b2198eca9fc2679666fe4ae1a8157a177bc6c53d3ff8425: Status 404 returned error can't find the container with id db27187ce40855919b2198eca9fc2679666fe4ae1a8157a177bc6c53d3ff8425
Jul 07 15:50:02.267007 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:02.266980    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5962e3e-69f6-42f4-b642-7bac15a37ede-tls-certs\") pod \"model-serving-api-54d89bd5fc-nkdqs\" (UID: \"e5962e3e-69f6-42f4-b642-7bac15a37ede\") " pod="opendatahub/model-serving-api-54d89bd5fc-nkdqs"
Jul 07 15:50:02.267174 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:02.267018    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsw9d\" (UniqueName: \"kubernetes.io/projected/e5962e3e-69f6-42f4-b642-7bac15a37ede-kube-api-access-nsw9d\") pod \"model-serving-api-54d89bd5fc-nkdqs\" (UID: \"e5962e3e-69f6-42f4-b642-7bac15a37ede\") " pod="opendatahub/model-serving-api-54d89bd5fc-nkdqs"
Jul 07 15:50:02.269574 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:02.269548    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5962e3e-69f6-42f4-b642-7bac15a37ede-tls-certs\") pod \"model-serving-api-54d89bd5fc-nkdqs\" (UID: \"e5962e3e-69f6-42f4-b642-7bac15a37ede\") " pod="opendatahub/model-serving-api-54d89bd5fc-nkdqs"
Jul 07 15:50:02.276694 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:02.276670    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsw9d\" (UniqueName: \"kubernetes.io/projected/e5962e3e-69f6-42f4-b642-7bac15a37ede-kube-api-access-nsw9d\") pod \"model-serving-api-54d89bd5fc-nkdqs\" (UID: \"e5962e3e-69f6-42f4-b642-7bac15a37ede\") " pod="opendatahub/model-serving-api-54d89bd5fc-nkdqs"
Jul 07 15:50:02.378195 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:02.378108    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-nkdqs"
Jul 07 15:50:02.535935 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:02.535904    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-nkdqs"]
Jul 07 15:50:02.538179 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:50:02.538152    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5962e3e_69f6_42f4_b642_7bac15a37ede.slice/crio-c61bb71c6bcfe010f81ad46cde23e0c81d57369d66a677faaeca920d3e32863b WatchSource:0}: Error finding container c61bb71c6bcfe010f81ad46cde23e0c81d57369d66a677faaeca920d3e32863b: Status 404 returned error can't find the container with id c61bb71c6bcfe010f81ad46cde23e0c81d57369d66a677faaeca920d3e32863b
Jul 07 15:50:02.789899 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:02.789844    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-nkdqs" event={"ID":"e5962e3e-69f6-42f4-b642-7bac15a37ede","Type":"ContainerStarted","Data":"c61bb71c6bcfe010f81ad46cde23e0c81d57369d66a677faaeca920d3e32863b"}
Jul 07 15:50:02.791352 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:02.791315    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-kt87x" event={"ID":"d1ce20f8-4500-4af3-b2e9-d5c6933df650","Type":"ContainerStarted","Data":"db27187ce40855919b2198eca9fc2679666fe4ae1a8157a177bc6c53d3ff8425"}
Jul 07 15:50:02.951717 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:02.951680    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-jx6xg"]
Jul 07 15:50:02.955346 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:02.955325    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-jx6xg"
Jul 07 15:50:02.957613 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:02.957590    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 07 15:50:02.957732 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:02.957611    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-txbh7\""
Jul 07 15:50:02.957732 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:02.957620    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 07 15:50:02.957732 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:02.957591    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 07 15:50:02.957732 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:02.957590    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 07 15:50:02.969690 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:02.969662    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-jx6xg"]
Jul 07 15:50:03.076434 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:03.076351    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b632b632-d0d4-42f2-ae52-2d60baf1338e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-jx6xg\" (UID: \"b632b632-d0d4-42f2-ae52-2d60baf1338e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-jx6xg"
Jul 07 15:50:03.076434 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:03.076419    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b632b632-d0d4-42f2-ae52-2d60baf1338e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-jx6xg\" (UID: \"b632b632-d0d4-42f2-ae52-2d60baf1338e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-jx6xg"
Jul 07 15:50:03.076669 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:03.076457    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b632b632-d0d4-42f2-ae52-2d60baf1338e-cacerts\") pod \"istiod-openshift-gateway-7574459656-jx6xg\" (UID: \"b632b632-d0d4-42f2-ae52-2d60baf1338e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-jx6xg"
Jul 07 15:50:03.076669 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:03.076560    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b632b632-d0d4-42f2-ae52-2d60baf1338e-istio-token\") pod \"istiod-openshift-gateway-7574459656-jx6xg\" (UID: \"b632b632-d0d4-42f2-ae52-2d60baf1338e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-jx6xg"
Jul 07 15:50:03.076669 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:03.076604    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b632b632-d0d4-42f2-ae52-2d60baf1338e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-jx6xg\" (UID: \"b632b632-d0d4-42f2-ae52-2d60baf1338e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-jx6xg"
Jul 07 15:50:03.076669 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:03.076650    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7c7wq\" (UniqueName: \"kubernetes.io/projected/b632b632-d0d4-42f2-ae52-2d60baf1338e-kube-api-access-7c7wq\") pod \"istiod-openshift-gateway-7574459656-jx6xg\" (UID: \"b632b632-d0d4-42f2-ae52-2d60baf1338e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-jx6xg"
Jul 07 15:50:03.076824 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:03.076740    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b632b632-d0d4-42f2-ae52-2d60baf1338e-local-certs\") pod \"istiod-openshift-gateway-7574459656-jx6xg\" (UID: \"b632b632-d0d4-42f2-ae52-2d60baf1338e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-jx6xg"
Jul 07 15:50:03.178129 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:03.178090    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b632b632-d0d4-42f2-ae52-2d60baf1338e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-jx6xg\" (UID: \"b632b632-d0d4-42f2-ae52-2d60baf1338e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-jx6xg"
Jul 07 15:50:03.178307 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:03.178167    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b632b632-d0d4-42f2-ae52-2d60baf1338e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-jx6xg\" (UID: \"b632b632-d0d4-42f2-ae52-2d60baf1338e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-jx6xg"
Jul 07 15:50:03.178307 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:03.178198    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b632b632-d0d4-42f2-ae52-2d60baf1338e-cacerts\") pod \"istiod-openshift-gateway-7574459656-jx6xg\" (UID: \"b632b632-d0d4-42f2-ae52-2d60baf1338e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-jx6xg"
Jul 07 15:50:03.178307 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:03.178241    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b632b632-d0d4-42f2-ae52-2d60baf1338e-istio-token\") pod \"istiod-openshift-gateway-7574459656-jx6xg\" (UID: \"b632b632-d0d4-42f2-ae52-2d60baf1338e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-jx6xg"
Jul 07 15:50:03.178307 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:03.178264    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b632b632-d0d4-42f2-ae52-2d60baf1338e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-jx6xg\" (UID: \"b632b632-d0d4-42f2-ae52-2d60baf1338e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-jx6xg"
Jul 07 15:50:03.178307 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:03.178293    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c7wq\" (UniqueName: \"kubernetes.io/projected/b632b632-d0d4-42f2-ae52-2d60baf1338e-kube-api-access-7c7wq\") pod \"istiod-openshift-gateway-7574459656-jx6xg\" (UID: \"b632b632-d0d4-42f2-ae52-2d60baf1338e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-jx6xg"
Jul 07 15:50:03.178554 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:03.178319    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b632b632-d0d4-42f2-ae52-2d60baf1338e-local-certs\") pod \"istiod-openshift-gateway-7574459656-jx6xg\" (UID: \"b632b632-d0d4-42f2-ae52-2d60baf1338e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-jx6xg"
Jul 07 15:50:03.179603 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:03.179317    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b632b632-d0d4-42f2-ae52-2d60baf1338e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-jx6xg\" (UID: \"b632b632-d0d4-42f2-ae52-2d60baf1338e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-jx6xg"
Jul 07 15:50:03.182671 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:03.182643    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b632b632-d0d4-42f2-ae52-2d60baf1338e-local-certs\") pod \"istiod-openshift-gateway-7574459656-jx6xg\" (UID: \"b632b632-d0d4-42f2-ae52-2d60baf1338e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-jx6xg"
Jul 07 15:50:03.183077 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:03.183057    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b632b632-d0d4-42f2-ae52-2d60baf1338e-cacerts\") pod \"istiod-openshift-gateway-7574459656-jx6xg\" (UID: \"b632b632-d0d4-42f2-ae52-2d60baf1338e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-jx6xg"
Jul 07 15:50:03.184457 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:03.184406    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b632b632-d0d4-42f2-ae52-2d60baf1338e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-jx6xg\" (UID: \"b632b632-d0d4-42f2-ae52-2d60baf1338e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-jx6xg"
Jul 07 15:50:03.187589 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:03.187529    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b632b632-d0d4-42f2-ae52-2d60baf1338e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-jx6xg\" (UID: \"b632b632-d0d4-42f2-ae52-2d60baf1338e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-jx6xg"
Jul 07 15:50:03.188162 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:03.188097    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b632b632-d0d4-42f2-ae52-2d60baf1338e-istio-token\") pod \"istiod-openshift-gateway-7574459656-jx6xg\" (UID: \"b632b632-d0d4-42f2-ae52-2d60baf1338e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-jx6xg"
Jul 07 15:50:03.194796 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:03.194771    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7c7wq\" (UniqueName: \"kubernetes.io/projected/b632b632-d0d4-42f2-ae52-2d60baf1338e-kube-api-access-7c7wq\") pod \"istiod-openshift-gateway-7574459656-jx6xg\" (UID: \"b632b632-d0d4-42f2-ae52-2d60baf1338e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-jx6xg"
Jul 07 15:50:03.278018 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:03.277977    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-jx6xg"
Jul 07 15:50:03.460166 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:03.460130    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-jx6xg"]
Jul 07 15:50:03.466520 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:50:03.466478    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb632b632_d0d4_42f2_ae52_2d60baf1338e.slice/crio-613146801e6fb77c52479cc180d810f7686842254ed344eb2591540eec74ff29 WatchSource:0}: Error finding container 613146801e6fb77c52479cc180d810f7686842254ed344eb2591540eec74ff29: Status 404 returned error can't find the container with id 613146801e6fb77c52479cc180d810f7686842254ed344eb2591540eec74ff29
Jul 07 15:50:03.799467 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:03.799388    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-jx6xg" event={"ID":"b632b632-d0d4-42f2-ae52-2d60baf1338e","Type":"ContainerStarted","Data":"613146801e6fb77c52479cc180d810f7686842254ed344eb2591540eec74ff29"}
Jul 07 15:50:06.494775 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:06.494747    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 07 15:50:07.453680 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:07.453645    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 07 15:50:07.453767 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:07.453709    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 07 15:50:07.820335 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:07.820297    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-59ffcfbcb9-n7k4p" event={"ID":"18a0d440-92e1-41b8-91ab-b9bb82c6d229","Type":"ContainerStarted","Data":"4b0c8dbea454027371d2af7b2a9c65438236e820508ed9d3ffa9c600ed6294a2"}
Jul 07 15:50:07.822226 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:07.822200    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-kt87x" event={"ID":"d1ce20f8-4500-4af3-b2e9-d5c6933df650","Type":"ContainerStarted","Data":"d0a98e83fbfc7c13af62b256181e8043d1b9afd7436719483308832e31cfff2a"}
Jul 07 15:50:07.822388 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:07.822302    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-kt87x"
Jul 07 15:50:07.823969 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:07.823911    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-nkdqs" event={"ID":"e5962e3e-69f6-42f4-b642-7bac15a37ede","Type":"ContainerStarted","Data":"964f1f36785e76d3cbb1501a1e5619d2e75bc0026c8349c7571889909bf11439"}
Jul 07 15:50:07.825607 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:07.825587    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-jx6xg" event={"ID":"b632b632-d0d4-42f2-ae52-2d60baf1338e","Type":"ContainerStarted","Data":"207854acdd5379f19091c4199082905ef8ff3277e6fc25fbc69e45f0c4157fcf"}
Jul 07 15:50:07.825717 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:07.825692    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-jx6xg"
Jul 07 15:50:07.841912 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:07.841837    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-59ffcfbcb9-n7k4p" podStartSLOduration=2.6153806939999997 podStartE2EDuration="7.841822374s" totalImagesPullingTime="5.22644168s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:50:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:50:07.841296074 +0000 UTC m=+361.929721745" watchObservedRunningTime="2026-07-07 15:50:07.841822374 +0000 UTC m=+361.930248044"
Jul 07 15:50:07.863758 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:07.863709    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-kt87x" podStartSLOduration=1.706535098 podStartE2EDuration="6.863691948s" totalImagesPullingTime="5.15715685s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:50:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:50:07.862227411 +0000 UTC m=+361.950653080" watchObservedRunningTime="2026-07-07 15:50:07.863691948 +0000 UTC m=+361.952117619"
Jul 07 15:50:07.883603 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:07.883540    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-nkdqs" podStartSLOduration=1.035310771 podStartE2EDuration="5.883518758s" totalImagesPullingTime="4.848207987s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:50:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:50:07.881701202 +0000 UTC m=+361.970126871" watchObservedRunningTime="2026-07-07 15:50:07.883518758 +0000 UTC m=+361.971944429"
Jul 07 15:50:07.907345 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:07.907290    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-jx6xg" podStartSLOduration=1.9242315730000001 podStartE2EDuration="5.907273597s" totalImagesPullingTime="3.983042024s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:50:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:50:07.906206481 +0000 UTC m=+361.994632151" watchObservedRunningTime="2026-07-07 15:50:07.907273597 +0000 UTC m=+361.995699267"
Jul 07 15:50:08.829832 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:08.829797    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-nkdqs"
Jul 07 15:50:08.831449 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:08.831419    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-jx6xg"
Jul 07 15:50:12.855756 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:12.855721    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-28lvw"]
Jul 07 15:50:12.859478 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:12.859454    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-28lvw"
Jul 07 15:50:12.860054 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:12.860032    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-b69vl"]
Jul 07 15:50:12.862834 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:12.862811    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Jul 07 15:50:12.863193 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:12.863170    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-b69vl"
Jul 07 15:50:12.863629 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:12.863612    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-xx4ft\""
Jul 07 15:50:12.865393 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:12.865378    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Jul 07 15:50:12.865600 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:12.865575    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-mv5hh\""
Jul 07 15:50:12.880700 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:12.880676    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-28lvw"]
Jul 07 15:50:12.889107 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:12.889078    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-b69vl"]
Jul 07 15:50:12.964533 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:12.964497    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bc47\" (UniqueName: \"kubernetes.io/projected/73a96c42-cb3f-4acf-adf4-31c21f32186c-kube-api-access-2bc47\") pod \"llmisvc-controller-manager-777c6bd7d5-b69vl\" (UID: \"73a96c42-cb3f-4acf-adf4-31c21f32186c\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-b69vl"
Jul 07 15:50:12.964533 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:12.964537    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/73a96c42-cb3f-4acf-adf4-31c21f32186c-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-b69vl\" (UID: \"73a96c42-cb3f-4acf-adf4-31c21f32186c\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-b69vl"
Jul 07 15:50:12.964770 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:12.964561    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cdc471c4-bece-4f05-9a9c-cf110a5eabb6-cert\") pod \"kserve-controller-manager-d6f4b65b8-28lvw\" (UID: \"cdc471c4-bece-4f05-9a9c-cf110a5eabb6\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-28lvw"
Jul 07 15:50:12.964770 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:12.964617    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkrln\" (UniqueName: \"kubernetes.io/projected/cdc471c4-bece-4f05-9a9c-cf110a5eabb6-kube-api-access-wkrln\") pod \"kserve-controller-manager-d6f4b65b8-28lvw\" (UID: \"cdc471c4-bece-4f05-9a9c-cf110a5eabb6\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-28lvw"
Jul 07 15:50:13.065459 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:13.065422    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bc47\" (UniqueName: \"kubernetes.io/projected/73a96c42-cb3f-4acf-adf4-31c21f32186c-kube-api-access-2bc47\") pod \"llmisvc-controller-manager-777c6bd7d5-b69vl\" (UID: \"73a96c42-cb3f-4acf-adf4-31c21f32186c\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-b69vl"
Jul 07 15:50:13.065459 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:13.065460    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/73a96c42-cb3f-4acf-adf4-31c21f32186c-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-b69vl\" (UID: \"73a96c42-cb3f-4acf-adf4-31c21f32186c\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-b69vl"
Jul 07 15:50:13.065649 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:13.065486    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cdc471c4-bece-4f05-9a9c-cf110a5eabb6-cert\") pod \"kserve-controller-manager-d6f4b65b8-28lvw\" (UID: \"cdc471c4-bece-4f05-9a9c-cf110a5eabb6\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-28lvw"
Jul 07 15:50:13.065649 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:13.065518    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkrln\" (UniqueName: \"kubernetes.io/projected/cdc471c4-bece-4f05-9a9c-cf110a5eabb6-kube-api-access-wkrln\") pod \"kserve-controller-manager-d6f4b65b8-28lvw\" (UID: \"cdc471c4-bece-4f05-9a9c-cf110a5eabb6\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-28lvw"
Jul 07 15:50:13.065649 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:50:13.065586    2566 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Jul 07 15:50:13.065755 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:50:13.065588    2566 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Jul 07 15:50:13.065755 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:50:13.065656    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73a96c42-cb3f-4acf-adf4-31c21f32186c-cert podName:73a96c42-cb3f-4acf-adf4-31c21f32186c nodeName:}" failed. No retries permitted until 2026-07-07 15:50:13.565635257 +0000 UTC m=+367.654060924 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/73a96c42-cb3f-4acf-adf4-31c21f32186c-cert") pod "llmisvc-controller-manager-777c6bd7d5-b69vl" (UID: "73a96c42-cb3f-4acf-adf4-31c21f32186c") : secret "llmisvc-webhook-server-cert" not found
Jul 07 15:50:13.065755 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:50:13.065723    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cdc471c4-bece-4f05-9a9c-cf110a5eabb6-cert podName:cdc471c4-bece-4f05-9a9c-cf110a5eabb6 nodeName:}" failed. No retries permitted until 2026-07-07 15:50:13.565708793 +0000 UTC m=+367.654134441 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/cdc471c4-bece-4f05-9a9c-cf110a5eabb6-cert") pod "kserve-controller-manager-d6f4b65b8-28lvw" (UID: "cdc471c4-bece-4f05-9a9c-cf110a5eabb6") : secret "kserve-webhook-server-cert" not found
Jul 07 15:50:13.080578 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:13.080532    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bc47\" (UniqueName: \"kubernetes.io/projected/73a96c42-cb3f-4acf-adf4-31c21f32186c-kube-api-access-2bc47\") pod \"llmisvc-controller-manager-777c6bd7d5-b69vl\" (UID: \"73a96c42-cb3f-4acf-adf4-31c21f32186c\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-b69vl"
Jul 07 15:50:13.081448 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:13.081426    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkrln\" (UniqueName: \"kubernetes.io/projected/cdc471c4-bece-4f05-9a9c-cf110a5eabb6-kube-api-access-wkrln\") pod \"kserve-controller-manager-d6f4b65b8-28lvw\" (UID: \"cdc471c4-bece-4f05-9a9c-cf110a5eabb6\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-28lvw"
Jul 07 15:50:13.569746 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:13.569698    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/73a96c42-cb3f-4acf-adf4-31c21f32186c-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-b69vl\" (UID: \"73a96c42-cb3f-4acf-adf4-31c21f32186c\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-b69vl"
Jul 07 15:50:13.569984 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:13.569822    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cdc471c4-bece-4f05-9a9c-cf110a5eabb6-cert\") pod \"kserve-controller-manager-d6f4b65b8-28lvw\" (UID: \"cdc471c4-bece-4f05-9a9c-cf110a5eabb6\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-28lvw"
Jul 07 15:50:13.572207 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:13.572180    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/73a96c42-cb3f-4acf-adf4-31c21f32186c-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-b69vl\" (UID: \"73a96c42-cb3f-4acf-adf4-31c21f32186c\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-b69vl"
Jul 07 15:50:13.572299 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:13.572286    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cdc471c4-bece-4f05-9a9c-cf110a5eabb6-cert\") pod \"kserve-controller-manager-d6f4b65b8-28lvw\" (UID: \"cdc471c4-bece-4f05-9a9c-cf110a5eabb6\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-28lvw"
Jul 07 15:50:13.773069 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:13.773032    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-28lvw"
Jul 07 15:50:13.781074 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:13.781048    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-b69vl"
Jul 07 15:50:13.936452 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:13.936415    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-28lvw"]
Jul 07 15:50:13.938485 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:50:13.938452    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdc471c4_bece_4f05_9a9c_cf110a5eabb6.slice/crio-bb875aeb9ca449f9b56c0f0cc1f426e9fcc1a1e1a4a1e284aa81c7297dca13a0 WatchSource:0}: Error finding container bb875aeb9ca449f9b56c0f0cc1f426e9fcc1a1e1a4a1e284aa81c7297dca13a0: Status 404 returned error can't find the container with id bb875aeb9ca449f9b56c0f0cc1f426e9fcc1a1e1a4a1e284aa81c7297dca13a0
Jul 07 15:50:13.947044 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:13.947016    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-b69vl"]
Jul 07 15:50:13.950113 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:50:13.950088    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod73a96c42_cb3f_4acf_adf4_31c21f32186c.slice/crio-40baa14f32d492395ac34d0b444fb6b049fae6096ae23de14955a8314efeaad7 WatchSource:0}: Error finding container 40baa14f32d492395ac34d0b444fb6b049fae6096ae23de14955a8314efeaad7: Status 404 returned error can't find the container with id 40baa14f32d492395ac34d0b444fb6b049fae6096ae23de14955a8314efeaad7
Jul 07 15:50:14.857210 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:14.857146    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-b69vl" event={"ID":"73a96c42-cb3f-4acf-adf4-31c21f32186c","Type":"ContainerStarted","Data":"40baa14f32d492395ac34d0b444fb6b049fae6096ae23de14955a8314efeaad7"}
Jul 07 15:50:14.858582 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:14.858554    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-28lvw" event={"ID":"cdc471c4-bece-4f05-9a9c-cf110a5eabb6","Type":"ContainerStarted","Data":"bb875aeb9ca449f9b56c0f0cc1f426e9fcc1a1e1a4a1e284aa81c7297dca13a0"}
Jul 07 15:50:15.724864 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:15.724826    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-jx6xg"]
Jul 07 15:50:15.725335 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:15.725128    2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-jx6xg" podUID="b632b632-d0d4-42f2-ae52-2d60baf1338e" containerName="discovery" containerID="cri-o://207854acdd5379f19091c4199082905ef8ff3277e6fc25fbc69e45f0c4157fcf" gracePeriod=30
Jul 07 15:50:15.866223 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:15.866179    2566 generic.go:356] "Generic (PLEG): container finished" podID="b632b632-d0d4-42f2-ae52-2d60baf1338e" containerID="207854acdd5379f19091c4199082905ef8ff3277e6fc25fbc69e45f0c4157fcf" exitCode=0
Jul 07 15:50:15.866393 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:15.866240    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-jx6xg" event={"ID":"b632b632-d0d4-42f2-ae52-2d60baf1338e","Type":"ContainerDied","Data":"207854acdd5379f19091c4199082905ef8ff3277e6fc25fbc69e45f0c4157fcf"}
Jul 07 15:50:16.482281 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:16.482099    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-jx6xg"
Jul 07 15:50:16.615629 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:16.615594    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b632b632-d0d4-42f2-ae52-2d60baf1338e-istio-csr-ca-configmap\") pod \"b632b632-d0d4-42f2-ae52-2d60baf1338e\" (UID: \"b632b632-d0d4-42f2-ae52-2d60baf1338e\") "
Jul 07 15:50:16.615812 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:16.615647    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b632b632-d0d4-42f2-ae52-2d60baf1338e-cacerts\") pod \"b632b632-d0d4-42f2-ae52-2d60baf1338e\" (UID: \"b632b632-d0d4-42f2-ae52-2d60baf1338e\") "
Jul 07 15:50:16.615812 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:16.615673    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b632b632-d0d4-42f2-ae52-2d60baf1338e-istio-kubeconfig\") pod \"b632b632-d0d4-42f2-ae52-2d60baf1338e\" (UID: \"b632b632-d0d4-42f2-ae52-2d60baf1338e\") "
Jul 07 15:50:16.615812 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:16.615708    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7c7wq\" (UniqueName: \"kubernetes.io/projected/b632b632-d0d4-42f2-ae52-2d60baf1338e-kube-api-access-7c7wq\") pod \"b632b632-d0d4-42f2-ae52-2d60baf1338e\" (UID: \"b632b632-d0d4-42f2-ae52-2d60baf1338e\") "
Jul 07 15:50:16.615812 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:16.615740    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b632b632-d0d4-42f2-ae52-2d60baf1338e-istio-csr-dns-cert\") pod \"b632b632-d0d4-42f2-ae52-2d60baf1338e\" (UID: \"b632b632-d0d4-42f2-ae52-2d60baf1338e\") "
Jul 07 15:50:16.615812 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:16.615768    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b632b632-d0d4-42f2-ae52-2d60baf1338e-istio-token\") pod \"b632b632-d0d4-42f2-ae52-2d60baf1338e\" (UID: \"b632b632-d0d4-42f2-ae52-2d60baf1338e\") "
Jul 07 15:50:16.615812 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:16.615795    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b632b632-d0d4-42f2-ae52-2d60baf1338e-local-certs\") pod \"b632b632-d0d4-42f2-ae52-2d60baf1338e\" (UID: \"b632b632-d0d4-42f2-ae52-2d60baf1338e\") "
Jul 07 15:50:16.616132 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:16.616083    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b632b632-d0d4-42f2-ae52-2d60baf1338e-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "b632b632-d0d4-42f2-ae52-2d60baf1338e" (UID: "b632b632-d0d4-42f2-ae52-2d60baf1338e"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 15:50:16.618763 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:16.618735    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b632b632-d0d4-42f2-ae52-2d60baf1338e-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "b632b632-d0d4-42f2-ae52-2d60baf1338e" (UID: "b632b632-d0d4-42f2-ae52-2d60baf1338e"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 15:50:16.618904 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:16.618793    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b632b632-d0d4-42f2-ae52-2d60baf1338e-istio-token" (OuterVolumeSpecName: "istio-token") pod "b632b632-d0d4-42f2-ae52-2d60baf1338e" (UID: "b632b632-d0d4-42f2-ae52-2d60baf1338e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 15:50:16.618904 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:16.618826    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b632b632-d0d4-42f2-ae52-2d60baf1338e-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "b632b632-d0d4-42f2-ae52-2d60baf1338e" (UID: "b632b632-d0d4-42f2-ae52-2d60baf1338e"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 15:50:16.618904 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:16.618880    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b632b632-d0d4-42f2-ae52-2d60baf1338e-kube-api-access-7c7wq" (OuterVolumeSpecName: "kube-api-access-7c7wq") pod "b632b632-d0d4-42f2-ae52-2d60baf1338e" (UID: "b632b632-d0d4-42f2-ae52-2d60baf1338e"). InnerVolumeSpecName "kube-api-access-7c7wq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 15:50:16.619076 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:16.618986    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b632b632-d0d4-42f2-ae52-2d60baf1338e-cacerts" (OuterVolumeSpecName: "cacerts") pod "b632b632-d0d4-42f2-ae52-2d60baf1338e" (UID: "b632b632-d0d4-42f2-ae52-2d60baf1338e"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 15:50:16.619076 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:16.619055    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b632b632-d0d4-42f2-ae52-2d60baf1338e-local-certs" (OuterVolumeSpecName: "local-certs") pod "b632b632-d0d4-42f2-ae52-2d60baf1338e" (UID: "b632b632-d0d4-42f2-ae52-2d60baf1338e"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 15:50:16.717308 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:16.717229    2566 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b632b632-d0d4-42f2-ae52-2d60baf1338e-cacerts\") on node \"ip-10-0-138-101.ec2.internal\" DevicePath \"\""
Jul 07 15:50:16.717308 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:16.717260    2566 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b632b632-d0d4-42f2-ae52-2d60baf1338e-istio-kubeconfig\") on node \"ip-10-0-138-101.ec2.internal\" DevicePath \"\""
Jul 07 15:50:16.717308 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:16.717277    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7c7wq\" (UniqueName: \"kubernetes.io/projected/b632b632-d0d4-42f2-ae52-2d60baf1338e-kube-api-access-7c7wq\") on node \"ip-10-0-138-101.ec2.internal\" DevicePath \"\""
Jul 07 15:50:16.717308 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:16.717294    2566 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b632b632-d0d4-42f2-ae52-2d60baf1338e-istio-csr-dns-cert\") on node \"ip-10-0-138-101.ec2.internal\" DevicePath \"\""
Jul 07 15:50:16.717308 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:16.717307    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b632b632-d0d4-42f2-ae52-2d60baf1338e-istio-token\") on node \"ip-10-0-138-101.ec2.internal\" DevicePath \"\""
Jul 07 15:50:16.717594 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:16.717318    2566 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b632b632-d0d4-42f2-ae52-2d60baf1338e-local-certs\") on node \"ip-10-0-138-101.ec2.internal\" DevicePath \"\""
Jul 07 15:50:16.717594 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:16.717326    2566 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b632b632-d0d4-42f2-ae52-2d60baf1338e-istio-csr-ca-configmap\") on node \"ip-10-0-138-101.ec2.internal\" DevicePath \"\""
Jul 07 15:50:16.872363 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:16.872316    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-jx6xg" event={"ID":"b632b632-d0d4-42f2-ae52-2d60baf1338e","Type":"ContainerDied","Data":"613146801e6fb77c52479cc180d810f7686842254ed344eb2591540eec74ff29"}
Jul 07 15:50:16.872817 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:16.872367    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-jx6xg"
Jul 07 15:50:16.872817 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:16.872393    2566 scope.go:117] "RemoveContainer" containerID="207854acdd5379f19091c4199082905ef8ff3277e6fc25fbc69e45f0c4157fcf"
Jul 07 15:50:16.898378 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:16.898340    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-jx6xg"]
Jul 07 15:50:16.902238 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:16.902212    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-jx6xg"]
Jul 07 15:50:17.877204 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:17.877167    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-b69vl" event={"ID":"73a96c42-cb3f-4acf-adf4-31c21f32186c","Type":"ContainerStarted","Data":"4bf9794bfd5aa3f81bd316e6d17284773a3fb1e55b302823786d3a93972e5cba"}
Jul 07 15:50:17.877668 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:17.877288    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-b69vl"
Jul 07 15:50:17.878785 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:17.878761    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-28lvw" event={"ID":"cdc471c4-bece-4f05-9a9c-cf110a5eabb6","Type":"ContainerStarted","Data":"4c2f89c4e0ef98770d94eebf9abe9dbe494a39b6200447aa92276a85110f1e61"}
Jul 07 15:50:17.878923 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:17.878851    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-28lvw"
Jul 07 15:50:17.896926 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:17.896850    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-b69vl" podStartSLOduration=2.790165822 podStartE2EDuration="5.896833811s" totalImagesPullingTime="3.106667989s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:50:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:50:17.895515778 +0000 UTC m=+371.983941448" watchObservedRunningTime="2026-07-07 15:50:17.896833811 +0000 UTC m=+371.985259480"
Jul 07 15:50:17.914804 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:17.914752    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-28lvw" podStartSLOduration=2.8000831330000002 podStartE2EDuration="5.914735334s" totalImagesPullingTime="3.114652201s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:50:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:50:17.91361771 +0000 UTC m=+372.002043380" watchObservedRunningTime="2026-07-07 15:50:17.914735334 +0000 UTC m=+372.003161002"
Jul 07 15:50:18.478332 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:18.478290    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b632b632-d0d4-42f2-ae52-2d60baf1338e" path="/var/lib/kubelet/pods/b632b632-d0d4-42f2-ae52-2d60baf1338e/volumes"
Jul 07 15:50:18.832698 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:18.832607    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-kt87x"
Jul 07 15:50:19.837947 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:19.837915    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-nkdqs"
Jul 07 15:50:48.886680 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:48.886645    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-b69vl"
Jul 07 15:50:48.889647 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:50:48.889624    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-28lvw"
Jul 07 15:51:01.849518 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:01.849486    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-gm8t6"]
Jul 07 15:51:01.849881 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:01.849800    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b632b632-d0d4-42f2-ae52-2d60baf1338e" containerName="discovery"
Jul 07 15:51:01.849881 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:01.849811    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b632b632-d0d4-42f2-ae52-2d60baf1338e" containerName="discovery"
Jul 07 15:51:01.849881 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:01.849861    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b632b632-d0d4-42f2-ae52-2d60baf1338e" containerName="discovery"
Jul 07 15:51:01.851667 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:01.851651    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-gm8t6"
Jul 07 15:51:01.853853 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:01.853827    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 07 15:51:01.853990 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:01.853862    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 07 15:51:01.853990 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:01.853963    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-d9h6x\""
Jul 07 15:51:01.869833 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:01.869803    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-gm8t6"]
Jul 07 15:51:01.952720 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:01.952685    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f25jj\" (UniqueName: \"kubernetes.io/projected/1e0cce07-796a-493b-bb2d-dd5802f6491b-kube-api-access-f25jj\") pod \"kuadrant-operator-catalog-gm8t6\" (UID: \"1e0cce07-796a-493b-bb2d-dd5802f6491b\") " pod="kuadrant-system/kuadrant-operator-catalog-gm8t6"
Jul 07 15:51:02.053551 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:02.053517    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f25jj\" (UniqueName: \"kubernetes.io/projected/1e0cce07-796a-493b-bb2d-dd5802f6491b-kube-api-access-f25jj\") pod \"kuadrant-operator-catalog-gm8t6\" (UID: \"1e0cce07-796a-493b-bb2d-dd5802f6491b\") " pod="kuadrant-system/kuadrant-operator-catalog-gm8t6"
Jul 07 15:51:02.062132 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:02.062100    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f25jj\" (UniqueName: \"kubernetes.io/projected/1e0cce07-796a-493b-bb2d-dd5802f6491b-kube-api-access-f25jj\") pod \"kuadrant-operator-catalog-gm8t6\" (UID: \"1e0cce07-796a-493b-bb2d-dd5802f6491b\") " pod="kuadrant-system/kuadrant-operator-catalog-gm8t6"
Jul 07 15:51:02.161386 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:02.161307    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-gm8t6"
Jul 07 15:51:02.215995 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:02.215961    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-gm8t6"]
Jul 07 15:51:02.296697 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:02.296674    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-gm8t6"]
Jul 07 15:51:02.299095 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:51:02.299067    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e0cce07_796a_493b_bb2d_dd5802f6491b.slice/crio-ad07b29dfd0438fb468b4f0fe4b6152b5ee93ed34112480510d5bbd67570d965 WatchSource:0}: Error finding container ad07b29dfd0438fb468b4f0fe4b6152b5ee93ed34112480510d5bbd67570d965: Status 404 returned error can't find the container with id ad07b29dfd0438fb468b4f0fe4b6152b5ee93ed34112480510d5bbd67570d965
Jul 07 15:51:02.425763 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:02.425676    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-8hf4r"]
Jul 07 15:51:02.428811 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:02.428788    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-8hf4r"
Jul 07 15:51:02.445087 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:02.445055    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-8hf4r"]
Jul 07 15:51:02.457437 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:02.457406    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mncd\" (UniqueName: \"kubernetes.io/projected/d933ecaf-43d5-4993-9851-793a315835b2-kube-api-access-2mncd\") pod \"kuadrant-operator-catalog-8hf4r\" (UID: \"d933ecaf-43d5-4993-9851-793a315835b2\") " pod="kuadrant-system/kuadrant-operator-catalog-8hf4r"
Jul 07 15:51:02.558237 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:02.558193    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mncd\" (UniqueName: \"kubernetes.io/projected/d933ecaf-43d5-4993-9851-793a315835b2-kube-api-access-2mncd\") pod \"kuadrant-operator-catalog-8hf4r\" (UID: \"d933ecaf-43d5-4993-9851-793a315835b2\") " pod="kuadrant-system/kuadrant-operator-catalog-8hf4r"
Jul 07 15:51:02.567379 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:02.567342    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mncd\" (UniqueName: \"kubernetes.io/projected/d933ecaf-43d5-4993-9851-793a315835b2-kube-api-access-2mncd\") pod \"kuadrant-operator-catalog-8hf4r\" (UID: \"d933ecaf-43d5-4993-9851-793a315835b2\") " pod="kuadrant-system/kuadrant-operator-catalog-8hf4r"
Jul 07 15:51:02.737418 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:02.737387    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-8hf4r"
Jul 07 15:51:02.891132 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:02.891086    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-8hf4r"]
Jul 07 15:51:02.924775 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:51:02.924738    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd933ecaf_43d5_4993_9851_793a315835b2.slice/crio-6b52b969eb331df9ba71ba28410d97700c5ee1dcdf1a6a047c1beb31ac907d34 WatchSource:0}: Error finding container 6b52b969eb331df9ba71ba28410d97700c5ee1dcdf1a6a047c1beb31ac907d34: Status 404 returned error can't find the container with id 6b52b969eb331df9ba71ba28410d97700c5ee1dcdf1a6a047c1beb31ac907d34
Jul 07 15:51:03.058140 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:03.058063    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-gm8t6" event={"ID":"1e0cce07-796a-493b-bb2d-dd5802f6491b","Type":"ContainerStarted","Data":"ad07b29dfd0438fb468b4f0fe4b6152b5ee93ed34112480510d5bbd67570d965"}
Jul 07 15:51:03.059183 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:03.059146    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-8hf4r" event={"ID":"d933ecaf-43d5-4993-9851-793a315835b2","Type":"ContainerStarted","Data":"6b52b969eb331df9ba71ba28410d97700c5ee1dcdf1a6a047c1beb31ac907d34"}
Jul 07 15:51:05.069779 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:05.069741    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-gm8t6" event={"ID":"1e0cce07-796a-493b-bb2d-dd5802f6491b","Type":"ContainerStarted","Data":"ebd6404a84c741d99050b5db68fd3cc9987f32bd4d3a73715e17c7433d8368bf"}
Jul 07 15:51:05.069779 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:05.069757    2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-gm8t6" podUID="1e0cce07-796a-493b-bb2d-dd5802f6491b" containerName="registry-server" containerID="cri-o://ebd6404a84c741d99050b5db68fd3cc9987f32bd4d3a73715e17c7433d8368bf" gracePeriod=2
Jul 07 15:51:05.071311 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:05.071286    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-8hf4r" event={"ID":"d933ecaf-43d5-4993-9851-793a315835b2","Type":"ContainerStarted","Data":"362559b84a8e3cf15d8ab52674ceda883d5f16d7e12b7499c3b0316000132371"}
Jul 07 15:51:05.091954 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:05.091902    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-gm8t6" podStartSLOduration=2.156433762 podStartE2EDuration="4.091868246s" totalImagesPullingTime="1.935434484s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:51:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:51:05.091134782 +0000 UTC m=+419.179560451" watchObservedRunningTime="2026-07-07 15:51:05.091868246 +0000 UTC m=+419.180293916"
Jul 07 15:51:05.110143 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:05.110092    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-8hf4r" podStartSLOduration=1.79783123 podStartE2EDuration="3.11007645s" totalImagesPullingTime="1.31224522s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:51:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:51:05.109480997 +0000 UTC m=+419.197906666" watchObservedRunningTime="2026-07-07 15:51:05.11007645 +0000 UTC m=+419.198502119"
Jul 07 15:51:05.308345 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:05.308318    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-gm8t6"
Jul 07 15:51:05.382185 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:05.382101    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f25jj\" (UniqueName: \"kubernetes.io/projected/1e0cce07-796a-493b-bb2d-dd5802f6491b-kube-api-access-f25jj\") pod \"1e0cce07-796a-493b-bb2d-dd5802f6491b\" (UID: \"1e0cce07-796a-493b-bb2d-dd5802f6491b\") "
Jul 07 15:51:05.384282 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:05.384255    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e0cce07-796a-493b-bb2d-dd5802f6491b-kube-api-access-f25jj" (OuterVolumeSpecName: "kube-api-access-f25jj") pod "1e0cce07-796a-493b-bb2d-dd5802f6491b" (UID: "1e0cce07-796a-493b-bb2d-dd5802f6491b"). InnerVolumeSpecName "kube-api-access-f25jj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 15:51:05.483300 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:05.483261    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f25jj\" (UniqueName: \"kubernetes.io/projected/1e0cce07-796a-493b-bb2d-dd5802f6491b-kube-api-access-f25jj\") on node \"ip-10-0-138-101.ec2.internal\" DevicePath \"\""
Jul 07 15:51:06.076736 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:06.076695    2566 generic.go:356] "Generic (PLEG): container finished" podID="1e0cce07-796a-493b-bb2d-dd5802f6491b" containerID="ebd6404a84c741d99050b5db68fd3cc9987f32bd4d3a73715e17c7433d8368bf" exitCode=0
Jul 07 15:51:06.077183 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:06.076744    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-gm8t6"
Jul 07 15:51:06.077183 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:06.076745    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-gm8t6" event={"ID":"1e0cce07-796a-493b-bb2d-dd5802f6491b","Type":"ContainerDied","Data":"ebd6404a84c741d99050b5db68fd3cc9987f32bd4d3a73715e17c7433d8368bf"}
Jul 07 15:51:06.077183 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:06.076795    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-gm8t6" event={"ID":"1e0cce07-796a-493b-bb2d-dd5802f6491b","Type":"ContainerDied","Data":"ad07b29dfd0438fb468b4f0fe4b6152b5ee93ed34112480510d5bbd67570d965"}
Jul 07 15:51:06.077183 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:06.076824    2566 scope.go:117] "RemoveContainer" containerID="ebd6404a84c741d99050b5db68fd3cc9987f32bd4d3a73715e17c7433d8368bf"
Jul 07 15:51:06.085518 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:06.085495    2566 scope.go:117] "RemoveContainer" containerID="ebd6404a84c741d99050b5db68fd3cc9987f32bd4d3a73715e17c7433d8368bf"
Jul 07 15:51:06.085769 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:51:06.085746    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebd6404a84c741d99050b5db68fd3cc9987f32bd4d3a73715e17c7433d8368bf\": container with ID starting with ebd6404a84c741d99050b5db68fd3cc9987f32bd4d3a73715e17c7433d8368bf not found: ID does not exist" containerID="ebd6404a84c741d99050b5db68fd3cc9987f32bd4d3a73715e17c7433d8368bf"
Jul 07 15:51:06.085823 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:06.085777    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebd6404a84c741d99050b5db68fd3cc9987f32bd4d3a73715e17c7433d8368bf"} err="failed to get container status \"ebd6404a84c741d99050b5db68fd3cc9987f32bd4d3a73715e17c7433d8368bf\": rpc error: code = NotFound desc = could not find container \"ebd6404a84c741d99050b5db68fd3cc9987f32bd4d3a73715e17c7433d8368bf\": container with ID starting with ebd6404a84c741d99050b5db68fd3cc9987f32bd4d3a73715e17c7433d8368bf not found: ID does not exist"
Jul 07 15:51:06.096499 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:06.096474    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-gm8t6"]
Jul 07 15:51:06.100200 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:06.100177    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-gm8t6"]
Jul 07 15:51:06.473497 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:06.473469    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e0cce07-796a-493b-bb2d-dd5802f6491b" path="/var/lib/kubelet/pods/1e0cce07-796a-493b-bb2d-dd5802f6491b/volumes"
Jul 07 15:51:12.738188 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:12.738128    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-8hf4r"
Jul 07 15:51:12.738590 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:12.738210    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-8hf4r"
Jul 07 15:51:12.760500 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:12.760465    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-8hf4r"
Jul 07 15:51:13.126741 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:13.126663    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-8hf4r"
Jul 07 15:51:33.766162 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:33.766076    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-ng7j5"]
Jul 07 15:51:33.766584 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:33.766367    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e0cce07-796a-493b-bb2d-dd5802f6491b" containerName="registry-server"
Jul 07 15:51:33.766584 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:33.766378    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e0cce07-796a-493b-bb2d-dd5802f6491b" containerName="registry-server"
Jul 07 15:51:33.766584 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:33.766423    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e0cce07-796a-493b-bb2d-dd5802f6491b" containerName="registry-server"
Jul 07 15:51:33.768217 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:33.768198    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-ng7j5"
Jul 07 15:51:33.770542 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:33.770518    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Jul 07 15:51:33.770655 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:33.770550    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-cng9p\""
Jul 07 15:51:33.784308 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:33.784279    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-ng7j5"]
Jul 07 15:51:33.914714 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:33.914678    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5qqn\" (UniqueName: \"kubernetes.io/projected/5320cfe0-479b-4ef9-bc2f-b3c368461783-kube-api-access-t5qqn\") pod \"dns-operator-controller-manager-864dd78c4c-ng7j5\" (UID: \"5320cfe0-479b-4ef9-bc2f-b3c368461783\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-ng7j5"
Jul 07 15:51:34.015591 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:34.015561    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5qqn\" (UniqueName: \"kubernetes.io/projected/5320cfe0-479b-4ef9-bc2f-b3c368461783-kube-api-access-t5qqn\") pod \"dns-operator-controller-manager-864dd78c4c-ng7j5\" (UID: \"5320cfe0-479b-4ef9-bc2f-b3c368461783\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-ng7j5"
Jul 07 15:51:34.029664 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:34.029604    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5qqn\" (UniqueName: \"kubernetes.io/projected/5320cfe0-479b-4ef9-bc2f-b3c368461783-kube-api-access-t5qqn\") pod \"dns-operator-controller-manager-864dd78c4c-ng7j5\" (UID: \"5320cfe0-479b-4ef9-bc2f-b3c368461783\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-ng7j5"
Jul 07 15:51:34.078463 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:34.078425    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-ng7j5"
Jul 07 15:51:34.207615 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:34.207586    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-ng7j5"]
Jul 07 15:51:34.210132 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:51:34.210101    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5320cfe0_479b_4ef9_bc2f_b3c368461783.slice/crio-769a2a05d2bd1d02fd6dd1850670043456458d334b0a95d4142f2be386630a64 WatchSource:0}: Error finding container 769a2a05d2bd1d02fd6dd1850670043456458d334b0a95d4142f2be386630a64: Status 404 returned error can't find the container with id 769a2a05d2bd1d02fd6dd1850670043456458d334b0a95d4142f2be386630a64
Jul 07 15:51:35.194541 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:35.194510    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-ng7j5" event={"ID":"5320cfe0-479b-4ef9-bc2f-b3c368461783","Type":"ContainerStarted","Data":"769a2a05d2bd1d02fd6dd1850670043456458d334b0a95d4142f2be386630a64"}
Jul 07 15:51:36.199916 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:36.199878    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-ng7j5" event={"ID":"5320cfe0-479b-4ef9-bc2f-b3c368461783","Type":"ContainerStarted","Data":"3f6a142f2f4cc2e9e67b72877148cb22301c64b45d5756da95364cfee0d8d80a"}
Jul 07 15:51:36.200301 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:36.200028    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-ng7j5"
Jul 07 15:51:36.221217 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:36.221165    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-ng7j5" podStartSLOduration=1.729054249 podStartE2EDuration="3.22114966s" totalImagesPullingTime="1.492095411s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:51:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:51:36.219771675 +0000 UTC m=+450.308197358" watchObservedRunningTime="2026-07-07 15:51:36.22114966 +0000 UTC m=+450.309575329"
Jul 07 15:51:37.456140 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:37.456092    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ldsk5"]
Jul 07 15:51:37.458582 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:37.458562    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ldsk5"
Jul 07 15:51:37.461283 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:37.461262    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-8khnt\""
Jul 07 15:51:37.473318 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:37.473289    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ldsk5"]
Jul 07 15:51:37.543513 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:37.543481    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67z76\" (UniqueName: \"kubernetes.io/projected/a8cdef93-c26f-492f-807f-dbe219e9d89c-kube-api-access-67z76\") pod \"limitador-operator-controller-manager-859d7b78bd-ldsk5\" (UID: \"a8cdef93-c26f-492f-807f-dbe219e9d89c\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ldsk5"
Jul 07 15:51:37.644107 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:37.644073    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67z76\" (UniqueName: \"kubernetes.io/projected/a8cdef93-c26f-492f-807f-dbe219e9d89c-kube-api-access-67z76\") pod \"limitador-operator-controller-manager-859d7b78bd-ldsk5\" (UID: \"a8cdef93-c26f-492f-807f-dbe219e9d89c\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ldsk5"
Jul 07 15:51:37.653046 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:37.653018    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67z76\" (UniqueName: \"kubernetes.io/projected/a8cdef93-c26f-492f-807f-dbe219e9d89c-kube-api-access-67z76\") pod \"limitador-operator-controller-manager-859d7b78bd-ldsk5\" (UID: \"a8cdef93-c26f-492f-807f-dbe219e9d89c\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ldsk5"
Jul 07 15:51:37.768858 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:37.768756    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ldsk5"
Jul 07 15:51:37.904054 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:37.904028    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ldsk5"]
Jul 07 15:51:37.906433 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:51:37.906392    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8cdef93_c26f_492f_807f_dbe219e9d89c.slice/crio-28c68ccf5b423fd95a5209fe2ab66b8e1c64e69faf4a2bd9d0f47a3978afa7a1 WatchSource:0}: Error finding container 28c68ccf5b423fd95a5209fe2ab66b8e1c64e69faf4a2bd9d0f47a3978afa7a1: Status 404 returned error can't find the container with id 28c68ccf5b423fd95a5209fe2ab66b8e1c64e69faf4a2bd9d0f47a3978afa7a1
Jul 07 15:51:38.209431 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:38.209400    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ldsk5" event={"ID":"a8cdef93-c26f-492f-807f-dbe219e9d89c","Type":"ContainerStarted","Data":"28c68ccf5b423fd95a5209fe2ab66b8e1c64e69faf4a2bd9d0f47a3978afa7a1"}
Jul 07 15:51:40.220964 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:40.220926    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ldsk5" event={"ID":"a8cdef93-c26f-492f-807f-dbe219e9d89c","Type":"ContainerStarted","Data":"50ebb601f746e0535b2cfa19e36429fceaa2da121b67cb63d85441ddb36a4f89"}
Jul 07 15:51:40.221391 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:40.221001    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ldsk5"
Jul 07 15:51:40.241763 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:40.241713    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ldsk5" podStartSLOduration=1.647821977 podStartE2EDuration="3.241697785s" totalImagesPullingTime="1.593875808s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:51:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:51:40.241267694 +0000 UTC m=+454.329693364" watchObservedRunningTime="2026-07-07 15:51:40.241697785 +0000 UTC m=+454.330123454"
Jul 07 15:51:42.056050 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:42.056015    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4p5cd"]
Jul 07 15:51:42.061698 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:42.061665    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4p5cd"
Jul 07 15:51:42.064186 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:42.064165    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-ttkqr\""
Jul 07 15:51:42.074778 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:42.074753    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4p5cd"]
Jul 07 15:51:42.182672 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:42.182638    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzxx4\" (UniqueName: \"kubernetes.io/projected/e2a44f7d-429b-46bb-aae0-bca959190050-kube-api-access-mzxx4\") pod \"kuadrant-operator-controller-manager-8684c9ff49-4p5cd\" (UID: \"e2a44f7d-429b-46bb-aae0-bca959190050\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4p5cd"
Jul 07 15:51:42.182808 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:42.182689    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e2a44f7d-429b-46bb-aae0-bca959190050-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-4p5cd\" (UID: \"e2a44f7d-429b-46bb-aae0-bca959190050\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4p5cd"
Jul 07 15:51:42.283472 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:42.283430    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e2a44f7d-429b-46bb-aae0-bca959190050-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-4p5cd\" (UID: \"e2a44f7d-429b-46bb-aae0-bca959190050\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4p5cd"
Jul 07 15:51:42.283629 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:42.283528    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzxx4\" (UniqueName: \"kubernetes.io/projected/e2a44f7d-429b-46bb-aae0-bca959190050-kube-api-access-mzxx4\") pod \"kuadrant-operator-controller-manager-8684c9ff49-4p5cd\" (UID: \"e2a44f7d-429b-46bb-aae0-bca959190050\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4p5cd"
Jul 07 15:51:42.283820 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:42.283801    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e2a44f7d-429b-46bb-aae0-bca959190050-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-4p5cd\" (UID: \"e2a44f7d-429b-46bb-aae0-bca959190050\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4p5cd"
Jul 07 15:51:42.292704 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:42.292675    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzxx4\" (UniqueName: \"kubernetes.io/projected/e2a44f7d-429b-46bb-aae0-bca959190050-kube-api-access-mzxx4\") pod \"kuadrant-operator-controller-manager-8684c9ff49-4p5cd\" (UID: \"e2a44f7d-429b-46bb-aae0-bca959190050\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4p5cd"
Jul 07 15:51:42.371396 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:42.371321    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4p5cd"
Jul 07 15:51:42.514646 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:42.514623    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4p5cd"]
Jul 07 15:51:42.517338 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:51:42.517316    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2a44f7d_429b_46bb_aae0_bca959190050.slice/crio-ff233b91e30aa8a79cea1c8b72e0cd3e737353b3961e5ca4331c785a17168acb WatchSource:0}: Error finding container ff233b91e30aa8a79cea1c8b72e0cd3e737353b3961e5ca4331c785a17168acb: Status 404 returned error can't find the container with id ff233b91e30aa8a79cea1c8b72e0cd3e737353b3961e5ca4331c785a17168acb
Jul 07 15:51:43.233738 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:43.233707    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4p5cd" event={"ID":"e2a44f7d-429b-46bb-aae0-bca959190050","Type":"ContainerStarted","Data":"ff233b91e30aa8a79cea1c8b72e0cd3e737353b3961e5ca4331c785a17168acb"}
Jul 07 15:51:46.180855 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:46.180813    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-8dwtd"]
Jul 07 15:51:46.184701 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:46.184672    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-8dwtd"
Jul 07 15:51:46.187113 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:46.187082    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-vgdfs\""
Jul 07 15:51:46.194848 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:46.194785    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-8dwtd"]
Jul 07 15:51:46.318749 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:46.318704    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjvtb\" (UniqueName: \"kubernetes.io/projected/9cbfb56e-0a7a-4f1f-8eb8-df1615d4bc55-kube-api-access-rjvtb\") pod \"authorino-operator-75cc8d5cb7-8dwtd\" (UID: \"9cbfb56e-0a7a-4f1f-8eb8-df1615d4bc55\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-8dwtd"
Jul 07 15:51:46.419794 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:46.419753    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjvtb\" (UniqueName: \"kubernetes.io/projected/9cbfb56e-0a7a-4f1f-8eb8-df1615d4bc55-kube-api-access-rjvtb\") pod \"authorino-operator-75cc8d5cb7-8dwtd\" (UID: \"9cbfb56e-0a7a-4f1f-8eb8-df1615d4bc55\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-8dwtd"
Jul 07 15:51:46.430071 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:46.430040    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjvtb\" (UniqueName: \"kubernetes.io/projected/9cbfb56e-0a7a-4f1f-8eb8-df1615d4bc55-kube-api-access-rjvtb\") pod \"authorino-operator-75cc8d5cb7-8dwtd\" (UID: \"9cbfb56e-0a7a-4f1f-8eb8-df1615d4bc55\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-8dwtd"
Jul 07 15:51:46.498364 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:46.498320    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-8dwtd"
Jul 07 15:51:47.206435 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:47.206412    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-8dwtd"]
Jul 07 15:51:47.206938 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:47.206904    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-ng7j5"
Jul 07 15:51:47.209264 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:51:47.209229    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cbfb56e_0a7a_4f1f_8eb8_df1615d4bc55.slice/crio-5494b37b3b052366b33fb4227916407c8841b20a500eca91aa64caff3a09f596 WatchSource:0}: Error finding container 5494b37b3b052366b33fb4227916407c8841b20a500eca91aa64caff3a09f596: Status 404 returned error can't find the container with id 5494b37b3b052366b33fb4227916407c8841b20a500eca91aa64caff3a09f596
Jul 07 15:51:47.253317 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:47.253278    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4p5cd" event={"ID":"e2a44f7d-429b-46bb-aae0-bca959190050","Type":"ContainerStarted","Data":"cd48ba75067c2d988b45ec2bca9d316c602963692f4ef0df465e05b89e031ad0"}
Jul 07 15:51:47.253470 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:47.253426    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4p5cd"
Jul 07 15:51:47.254588 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:47.254557    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-8dwtd" event={"ID":"9cbfb56e-0a7a-4f1f-8eb8-df1615d4bc55","Type":"ContainerStarted","Data":"5494b37b3b052366b33fb4227916407c8841b20a500eca91aa64caff3a09f596"}
Jul 07 15:51:47.279584 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:47.279522    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4p5cd" podStartSLOduration=0.666245681 podStartE2EDuration="5.279500332s" totalImagesPullingTime="4.613254651s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:51:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:51:47.27829916 +0000 UTC m=+461.366724856" watchObservedRunningTime="2026-07-07 15:51:47.279500332 +0000 UTC m=+461.367926004"
Jul 07 15:51:50.273517 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:50.273480    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-8dwtd" event={"ID":"9cbfb56e-0a7a-4f1f-8eb8-df1615d4bc55","Type":"ContainerStarted","Data":"8edd6f634b8414a18dd046ee5396af7f2aba963c4664749a0246aad9373e1539"}
Jul 07 15:51:50.273956 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:50.273571    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-8dwtd"
Jul 07 15:51:50.290647 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:50.290599    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-8dwtd" podStartSLOduration=2.134080532 podStartE2EDuration="4.290583166s" totalImagesPullingTime="2.156502634s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:51:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:51:50.289700753 +0000 UTC m=+464.378126425" watchObservedRunningTime="2026-07-07 15:51:50.290583166 +0000 UTC m=+464.379008836"
Jul 07 15:51:51.226956 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:51.226919    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ldsk5"
Jul 07 15:51:58.262877 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:58.262843    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4p5cd"
Jul 07 15:51:59.963082 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:59.963048    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4p5cd"]
Jul 07 15:51:59.963514 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:59.963269    2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4p5cd" podUID="e2a44f7d-429b-46bb-aae0-bca959190050" containerName="manager" containerID="cri-o://cd48ba75067c2d988b45ec2bca9d316c602963692f4ef0df465e05b89e031ad0" gracePeriod=2
Jul 07 15:51:59.973139 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:59.973114    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4p5cd"]
Jul 07 15:51:59.977427 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:51:59.977394    2566 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"e2a44f7d-429b-46bb-aae0-bca959190050\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-4p5cd\": pods \"kuadrant-operator-controller-manager-8684c9ff49-4p5cd\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4p5cd"
Jul 07 15:51:59.994710 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:59.994680    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ldsk5"]
Jul 07 15:51:59.994960 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:59.994935    2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ldsk5" podUID="a8cdef93-c26f-492f-807f-dbe219e9d89c" containerName="manager" containerID="cri-o://50ebb601f746e0535b2cfa19e36429fceaa2da121b67cb63d85441ddb36a4f89" gracePeriod=2
Jul 07 15:51:59.996391 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:59.996368    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-swqvw"]
Jul 07 15:51:59.996741 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:59.996722    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2a44f7d-429b-46bb-aae0-bca959190050" containerName="manager"
Jul 07 15:51:59.996813 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:59.996743    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2a44f7d-429b-46bb-aae0-bca959190050" containerName="manager"
Jul 07 15:51:59.996872 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:59.996856    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2a44f7d-429b-46bb-aae0-bca959190050" containerName="manager"
Jul 07 15:51:59.999806 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:51:59.999784    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-swqvw"
Jul 07 15:52:00.008102 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:00.008074    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ldsk5"]
Jul 07 15:52:00.013737 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:52:00.013705    2566 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"a8cdef93-c26f-492f-807f-dbe219e9d89c\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-859d7b78bd-ldsk5\": pods \"limitador-operator-controller-manager-859d7b78bd-ldsk5\" not found" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ldsk5"
Jul 07 15:52:00.016029 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:52:00.016000    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-4p5cd\" is forbidden: User \"system:node:ip-10-0-138-101.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-101.ec2.internal' and this object" podUID="e2a44f7d-429b-46bb-aae0-bca959190050" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4p5cd"
Jul 07 15:52:00.055971 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:00.055941    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-swqvw"]
Jul 07 15:52:00.074804 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:00.074777    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lg2l4"]
Jul 07 15:52:00.075224 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:00.075195    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8cdef93-c26f-492f-807f-dbe219e9d89c" containerName="manager"
Jul 07 15:52:00.075224 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:00.075226    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8cdef93-c26f-492f-807f-dbe219e9d89c" containerName="manager"
Jul 07 15:52:00.075376 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:00.075309    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8cdef93-c26f-492f-807f-dbe219e9d89c" containerName="manager"
Jul 07 15:52:00.076285 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:52:00.076257    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-ldsk5\" is forbidden: User \"system:node:ip-10-0-138-101.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-101.ec2.internal' and this object" podUID="a8cdef93-c26f-492f-807f-dbe219e9d89c" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ldsk5"
Jul 07 15:52:00.078025 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:00.078010    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lg2l4"
Jul 07 15:52:00.078198 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:52:00.078177    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-4p5cd\" is forbidden: User \"system:node:ip-10-0-138-101.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-101.ec2.internal' and this object" podUID="e2a44f7d-429b-46bb-aae0-bca959190050" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4p5cd"
Jul 07 15:52:00.096128 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:00.096100    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lg2l4"]
Jul 07 15:52:00.113120 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:52:00.113089    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-ldsk5\" is forbidden: User \"system:node:ip-10-0-138-101.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-101.ec2.internal' and this object" podUID="a8cdef93-c26f-492f-807f-dbe219e9d89c" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ldsk5"
Jul 07 15:52:00.115324 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:52:00.115269    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-4p5cd\" is forbidden: User \"system:node:ip-10-0-138-101.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-101.ec2.internal' and this object" podUID="e2a44f7d-429b-46bb-aae0-bca959190050" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4p5cd"
Jul 07 15:52:00.128661 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:00.128629    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5d6ec269-6c2a-4fe8-9e12-5a4315a9535e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-swqvw\" (UID: \"5d6ec269-6c2a-4fe8-9e12-5a4315a9535e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-swqvw"
Jul 07 15:52:00.128826 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:00.128687    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6qwn\" (UniqueName: \"kubernetes.io/projected/5d6ec269-6c2a-4fe8-9e12-5a4315a9535e-kube-api-access-j6qwn\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-swqvw\" (UID: \"5d6ec269-6c2a-4fe8-9e12-5a4315a9535e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-swqvw"
Jul 07 15:52:00.229146 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:00.229118    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6qwn\" (UniqueName: \"kubernetes.io/projected/5d6ec269-6c2a-4fe8-9e12-5a4315a9535e-kube-api-access-j6qwn\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-swqvw\" (UID: \"5d6ec269-6c2a-4fe8-9e12-5a4315a9535e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-swqvw"
Jul 07 15:52:00.229260 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:00.229186    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pds2l\" (UniqueName: \"kubernetes.io/projected/080269f8-421c-4ca6-bc67-ad9b974c6ca2-kube-api-access-pds2l\") pod \"limitador-operator-controller-manager-859d7b78bd-lg2l4\" (UID: \"080269f8-421c-4ca6-bc67-ad9b974c6ca2\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lg2l4"
Jul 07 15:52:00.229260 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:00.229246    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5d6ec269-6c2a-4fe8-9e12-5a4315a9535e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-swqvw\" (UID: \"5d6ec269-6c2a-4fe8-9e12-5a4315a9535e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-swqvw"
Jul 07 15:52:00.229548 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:00.229525    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5d6ec269-6c2a-4fe8-9e12-5a4315a9535e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-swqvw\" (UID: \"5d6ec269-6c2a-4fe8-9e12-5a4315a9535e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-swqvw"
Jul 07 15:52:00.237580 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:00.237548    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6qwn\" (UniqueName: \"kubernetes.io/projected/5d6ec269-6c2a-4fe8-9e12-5a4315a9535e-kube-api-access-j6qwn\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-swqvw\" (UID: \"5d6ec269-6c2a-4fe8-9e12-5a4315a9535e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-swqvw"
Jul 07 15:52:00.242742 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:00.242717    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4p5cd"
Jul 07 15:52:00.244953 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:52:00.244929    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-ldsk5\" is forbidden: User \"system:node:ip-10-0-138-101.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-101.ec2.internal' and this object" podUID="a8cdef93-c26f-492f-807f-dbe219e9d89c" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ldsk5"
Jul 07 15:52:00.245310 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:00.245291    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ldsk5"
Jul 07 15:52:00.246840 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:52:00.246820    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-4p5cd\" is forbidden: User \"system:node:ip-10-0-138-101.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-101.ec2.internal' and this object" podUID="e2a44f7d-429b-46bb-aae0-bca959190050" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4p5cd"
Jul 07 15:52:00.248498 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:52:00.248473    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-ldsk5\" is forbidden: User \"system:node:ip-10-0-138-101.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-101.ec2.internal' and this object" podUID="a8cdef93-c26f-492f-807f-dbe219e9d89c" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ldsk5"
Jul 07 15:52:00.250296 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:52:00.250278    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-4p5cd\" is forbidden: User \"system:node:ip-10-0-138-101.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-101.ec2.internal' and this object" podUID="e2a44f7d-429b-46bb-aae0-bca959190050" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4p5cd"
Jul 07 15:52:00.317869 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:00.317836    2566 generic.go:356] "Generic (PLEG): container finished" podID="a8cdef93-c26f-492f-807f-dbe219e9d89c" containerID="50ebb601f746e0535b2cfa19e36429fceaa2da121b67cb63d85441ddb36a4f89" exitCode=0
Jul 07 15:52:00.318031 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:00.317877    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ldsk5"
Jul 07 15:52:00.318031 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:00.317932    2566 scope.go:117] "RemoveContainer" containerID="50ebb601f746e0535b2cfa19e36429fceaa2da121b67cb63d85441ddb36a4f89"
Jul 07 15:52:00.319240 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:00.319218    2566 generic.go:356] "Generic (PLEG): container finished" podID="e2a44f7d-429b-46bb-aae0-bca959190050" containerID="cd48ba75067c2d988b45ec2bca9d316c602963692f4ef0df465e05b89e031ad0" exitCode=0
Jul 07 15:52:00.319331 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:00.319258    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4p5cd"
Jul 07 15:52:00.320269 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:52:00.320238    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-ldsk5\" is forbidden: User \"system:node:ip-10-0-138-101.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-101.ec2.internal' and this object" podUID="a8cdef93-c26f-492f-807f-dbe219e9d89c" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ldsk5"
Jul 07 15:52:00.321910 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:52:00.321875    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-4p5cd\" is forbidden: User \"system:node:ip-10-0-138-101.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-101.ec2.internal' and this object" podUID="e2a44f7d-429b-46bb-aae0-bca959190050" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4p5cd"
Jul 07 15:52:00.323424 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:52:00.323401    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-ldsk5\" is forbidden: User \"system:node:ip-10-0-138-101.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-101.ec2.internal' and this object" podUID="a8cdef93-c26f-492f-807f-dbe219e9d89c" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ldsk5"
Jul 07 15:52:00.325021 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:52:00.325000    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-4p5cd\" is forbidden: User \"system:node:ip-10-0-138-101.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-101.ec2.internal' and this object" podUID="e2a44f7d-429b-46bb-aae0-bca959190050" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4p5cd"
Jul 07 15:52:00.328148 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:00.328126    2566 scope.go:117] "RemoveContainer" containerID="50ebb601f746e0535b2cfa19e36429fceaa2da121b67cb63d85441ddb36a4f89"
Jul 07 15:52:00.328498 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:52:00.328477    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50ebb601f746e0535b2cfa19e36429fceaa2da121b67cb63d85441ddb36a4f89\": container with ID starting with 50ebb601f746e0535b2cfa19e36429fceaa2da121b67cb63d85441ddb36a4f89 not found: ID does not exist" containerID="50ebb601f746e0535b2cfa19e36429fceaa2da121b67cb63d85441ddb36a4f89"
Jul 07 15:52:00.328586 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:00.328505    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50ebb601f746e0535b2cfa19e36429fceaa2da121b67cb63d85441ddb36a4f89"} err="failed to get container status \"50ebb601f746e0535b2cfa19e36429fceaa2da121b67cb63d85441ddb36a4f89\": rpc error: code = NotFound desc = could not find container \"50ebb601f746e0535b2cfa19e36429fceaa2da121b67cb63d85441ddb36a4f89\": container with ID starting with 50ebb601f746e0535b2cfa19e36429fceaa2da121b67cb63d85441ddb36a4f89 not found: ID does not exist"
Jul 07 15:52:00.328586 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:00.328523    2566 scope.go:117] "RemoveContainer" containerID="cd48ba75067c2d988b45ec2bca9d316c602963692f4ef0df465e05b89e031ad0"
Jul 07 15:52:00.329610 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:00.329594    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pds2l\" (UniqueName: \"kubernetes.io/projected/080269f8-421c-4ca6-bc67-ad9b974c6ca2-kube-api-access-pds2l\") pod \"limitador-operator-controller-manager-859d7b78bd-lg2l4\" (UID: \"080269f8-421c-4ca6-bc67-ad9b974c6ca2\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lg2l4"
Jul 07 15:52:00.336931 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:00.336910    2566 scope.go:117] "RemoveContainer" containerID="cd48ba75067c2d988b45ec2bca9d316c602963692f4ef0df465e05b89e031ad0"
Jul 07 15:52:00.337232 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:52:00.337214    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd48ba75067c2d988b45ec2bca9d316c602963692f4ef0df465e05b89e031ad0\": container with ID starting with cd48ba75067c2d988b45ec2bca9d316c602963692f4ef0df465e05b89e031ad0 not found: ID does not exist" containerID="cd48ba75067c2d988b45ec2bca9d316c602963692f4ef0df465e05b89e031ad0"
Jul 07 15:52:00.337276 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:00.337238    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd48ba75067c2d988b45ec2bca9d316c602963692f4ef0df465e05b89e031ad0"} err="failed to get container status \"cd48ba75067c2d988b45ec2bca9d316c602963692f4ef0df465e05b89e031ad0\": rpc error: code = NotFound desc = could not find container \"cd48ba75067c2d988b45ec2bca9d316c602963692f4ef0df465e05b89e031ad0\": container with ID starting with cd48ba75067c2d988b45ec2bca9d316c602963692f4ef0df465e05b89e031ad0 not found: ID does not exist"
Jul 07 15:52:00.345051 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:00.345021    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pds2l\" (UniqueName: \"kubernetes.io/projected/080269f8-421c-4ca6-bc67-ad9b974c6ca2-kube-api-access-pds2l\") pod \"limitador-operator-controller-manager-859d7b78bd-lg2l4\" (UID: \"080269f8-421c-4ca6-bc67-ad9b974c6ca2\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lg2l4"
Jul 07 15:52:00.388779 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:00.388742    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-swqvw"
Jul 07 15:52:00.394832 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:00.394810    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lg2l4"
Jul 07 15:52:00.431179 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:00.430358    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mzxx4\" (UniqueName: \"kubernetes.io/projected/e2a44f7d-429b-46bb-aae0-bca959190050-kube-api-access-mzxx4\") pod \"e2a44f7d-429b-46bb-aae0-bca959190050\" (UID: \"e2a44f7d-429b-46bb-aae0-bca959190050\") "
Jul 07 15:52:00.431179 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:00.430445    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-67z76\" (UniqueName: \"kubernetes.io/projected/a8cdef93-c26f-492f-807f-dbe219e9d89c-kube-api-access-67z76\") pod \"a8cdef93-c26f-492f-807f-dbe219e9d89c\" (UID: \"a8cdef93-c26f-492f-807f-dbe219e9d89c\") "
Jul 07 15:52:00.431179 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:00.430494    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e2a44f7d-429b-46bb-aae0-bca959190050-extensions-socket-volume\") pod \"e2a44f7d-429b-46bb-aae0-bca959190050\" (UID: \"e2a44f7d-429b-46bb-aae0-bca959190050\") "
Jul 07 15:52:00.431490 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:00.431276    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2a44f7d-429b-46bb-aae0-bca959190050-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "e2a44f7d-429b-46bb-aae0-bca959190050" (UID: "e2a44f7d-429b-46bb-aae0-bca959190050"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 15:52:00.433249 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:00.433206    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2a44f7d-429b-46bb-aae0-bca959190050-kube-api-access-mzxx4" (OuterVolumeSpecName: "kube-api-access-mzxx4") pod "e2a44f7d-429b-46bb-aae0-bca959190050" (UID: "e2a44f7d-429b-46bb-aae0-bca959190050"). InnerVolumeSpecName "kube-api-access-mzxx4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 15:52:00.433374 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:00.433336    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8cdef93-c26f-492f-807f-dbe219e9d89c-kube-api-access-67z76" (OuterVolumeSpecName: "kube-api-access-67z76") pod "a8cdef93-c26f-492f-807f-dbe219e9d89c" (UID: "a8cdef93-c26f-492f-807f-dbe219e9d89c"). InnerVolumeSpecName "kube-api-access-67z76". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 15:52:00.474252 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:00.474168    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8cdef93-c26f-492f-807f-dbe219e9d89c" path="/var/lib/kubelet/pods/a8cdef93-c26f-492f-807f-dbe219e9d89c/volumes"
Jul 07 15:52:00.474556 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:00.474540    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2a44f7d-429b-46bb-aae0-bca959190050" path="/var/lib/kubelet/pods/e2a44f7d-429b-46bb-aae0-bca959190050/volumes"
Jul 07 15:52:00.531504 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:00.531469    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67z76\" (UniqueName: \"kubernetes.io/projected/a8cdef93-c26f-492f-807f-dbe219e9d89c-kube-api-access-67z76\") on node \"ip-10-0-138-101.ec2.internal\" DevicePath \"\""
Jul 07 15:52:00.531504 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:00.531498    2566 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e2a44f7d-429b-46bb-aae0-bca959190050-extensions-socket-volume\") on node \"ip-10-0-138-101.ec2.internal\" DevicePath \"\""
Jul 07 15:52:00.531504 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:00.531507    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzxx4\" (UniqueName: \"kubernetes.io/projected/e2a44f7d-429b-46bb-aae0-bca959190050-kube-api-access-mzxx4\") on node \"ip-10-0-138-101.ec2.internal\" DevicePath \"\""
Jul 07 15:52:00.538513 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:00.538488    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lg2l4"]
Jul 07 15:52:00.540836 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:52:00.540810    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod080269f8_421c_4ca6_bc67_ad9b974c6ca2.slice/crio-5bb9b01a4816a77f3ee5c6113289e40a51bd15902f7eccc4515c516c1c54bc40 WatchSource:0}: Error finding container 5bb9b01a4816a77f3ee5c6113289e40a51bd15902f7eccc4515c516c1c54bc40: Status 404 returned error can't find the container with id 5bb9b01a4816a77f3ee5c6113289e40a51bd15902f7eccc4515c516c1c54bc40
Jul 07 15:52:00.561170 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:00.561140    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-swqvw"]
Jul 07 15:52:00.564398 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:52:00.564366    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d6ec269_6c2a_4fe8_9e12_5a4315a9535e.slice/crio-e0bc608bf8630b40dbe9ea8de44484e2195778f5ed3012dd947107ff7b49e192 WatchSource:0}: Error finding container e0bc608bf8630b40dbe9ea8de44484e2195778f5ed3012dd947107ff7b49e192: Status 404 returned error can't find the container with id e0bc608bf8630b40dbe9ea8de44484e2195778f5ed3012dd947107ff7b49e192
Jul 07 15:52:01.279901 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:01.279802    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-8dwtd"
Jul 07 15:52:01.333540 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:01.333499    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-swqvw" event={"ID":"5d6ec269-6c2a-4fe8-9e12-5a4315a9535e","Type":"ContainerStarted","Data":"e8480d607c68aa1ffd0f1876ac96fa3044a04fa9095cf111b98d7bb10150eee8"}
Jul 07 15:52:01.333732 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:01.333594    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-swqvw" event={"ID":"5d6ec269-6c2a-4fe8-9e12-5a4315a9535e","Type":"ContainerStarted","Data":"e0bc608bf8630b40dbe9ea8de44484e2195778f5ed3012dd947107ff7b49e192"}
Jul 07 15:52:01.333732 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:01.333625    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-swqvw"
Jul 07 15:52:01.335220 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:01.335187    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lg2l4" event={"ID":"080269f8-421c-4ca6-bc67-ad9b974c6ca2","Type":"ContainerStarted","Data":"bf323bfd8eaa2ef32dd8bfcdd2cfc23d6445b9ed825295519dce6100d39b10a6"}
Jul 07 15:52:01.335348 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:01.335237    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lg2l4" event={"ID":"080269f8-421c-4ca6-bc67-ad9b974c6ca2","Type":"ContainerStarted","Data":"5bb9b01a4816a77f3ee5c6113289e40a51bd15902f7eccc4515c516c1c54bc40"}
Jul 07 15:52:01.335348 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:01.335305    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lg2l4"
Jul 07 15:52:01.354223 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:01.354013    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-swqvw" podStartSLOduration=2.353994043 podStartE2EDuration="2.353994043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:51:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:52:01.352986774 +0000 UTC m=+475.441412445" watchObservedRunningTime="2026-07-07 15:52:01.353994043 +0000 UTC m=+475.442419717"
Jul 07 15:52:01.373675 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:01.373615    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lg2l4" podStartSLOduration=1.373599397 podStartE2EDuration="1.373599397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:52:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:52:01.372516565 +0000 UTC m=+475.460942236" watchObservedRunningTime="2026-07-07 15:52:01.373599397 +0000 UTC m=+475.462025068"
Jul 07 15:52:12.343861 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:12.343826    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lg2l4"
Jul 07 15:52:12.344292 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:12.343929    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-swqvw"
Jul 07 15:52:12.742836 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:12.742806    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gvzb7"]
Jul 07 15:52:12.749082 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:12.749061    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gvzb7"
Jul 07 15:52:12.767010 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:12.766986    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gvzb7"]
Jul 07 15:52:12.820191 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:12.820160    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cc5263db-4499-40f7-8f0b-80f81d5e3a18-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-gvzb7\" (UID: \"cc5263db-4499-40f7-8f0b-80f81d5e3a18\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gvzb7"
Jul 07 15:52:12.820381 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:12.820204    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcts4\" (UniqueName: \"kubernetes.io/projected/cc5263db-4499-40f7-8f0b-80f81d5e3a18-kube-api-access-tcts4\") pod \"kuadrant-operator-controller-manager-86878c769f-gvzb7\" (UID: \"cc5263db-4499-40f7-8f0b-80f81d5e3a18\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gvzb7"
Jul 07 15:52:12.920875 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:12.920825    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcts4\" (UniqueName: \"kubernetes.io/projected/cc5263db-4499-40f7-8f0b-80f81d5e3a18-kube-api-access-tcts4\") pod \"kuadrant-operator-controller-manager-86878c769f-gvzb7\" (UID: \"cc5263db-4499-40f7-8f0b-80f81d5e3a18\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gvzb7"
Jul 07 15:52:12.921071 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:12.920942    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cc5263db-4499-40f7-8f0b-80f81d5e3a18-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-gvzb7\" (UID: \"cc5263db-4499-40f7-8f0b-80f81d5e3a18\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gvzb7"
Jul 07 15:52:12.921268 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:12.921252    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cc5263db-4499-40f7-8f0b-80f81d5e3a18-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-gvzb7\" (UID: \"cc5263db-4499-40f7-8f0b-80f81d5e3a18\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gvzb7"
Jul 07 15:52:12.935591 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:12.935561    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcts4\" (UniqueName: \"kubernetes.io/projected/cc5263db-4499-40f7-8f0b-80f81d5e3a18-kube-api-access-tcts4\") pod \"kuadrant-operator-controller-manager-86878c769f-gvzb7\" (UID: \"cc5263db-4499-40f7-8f0b-80f81d5e3a18\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gvzb7"
Jul 07 15:52:13.059821 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:13.059714    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gvzb7"
Jul 07 15:52:13.227084 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:13.227052    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gvzb7"]
Jul 07 15:52:13.230942 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:52:13.230907    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc5263db_4499_40f7_8f0b_80f81d5e3a18.slice/crio-63d0f200e720c331edad29033b8ef9f42db63789ffa4b8c762ae8e6e9c167e2f WatchSource:0}: Error finding container 63d0f200e720c331edad29033b8ef9f42db63789ffa4b8c762ae8e6e9c167e2f: Status 404 returned error can't find the container with id 63d0f200e720c331edad29033b8ef9f42db63789ffa4b8c762ae8e6e9c167e2f
Jul 07 15:52:13.395438 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:13.395354    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gvzb7" event={"ID":"cc5263db-4499-40f7-8f0b-80f81d5e3a18","Type":"ContainerStarted","Data":"02444f6ec83e22dee4186c943d6cd16b865d112a58dcb4c6d41fd711f0561da9"}
Jul 07 15:52:13.395438 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:13.395404    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gvzb7" event={"ID":"cc5263db-4499-40f7-8f0b-80f81d5e3a18","Type":"ContainerStarted","Data":"63d0f200e720c331edad29033b8ef9f42db63789ffa4b8c762ae8e6e9c167e2f"}
Jul 07 15:52:13.395864 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:13.395516    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gvzb7"
Jul 07 15:52:13.415517 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:13.415420    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gvzb7" podStartSLOduration=1.415402553 podStartE2EDuration="1.415402553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:52:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:52:13.415277965 +0000 UTC m=+487.503703637" watchObservedRunningTime="2026-07-07 15:52:13.415402553 +0000 UTC m=+487.503828224"
Jul 07 15:52:24.401433 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:24.401398    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gvzb7"
Jul 07 15:52:24.512458 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:24.512423    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-swqvw"]
Jul 07 15:52:24.512649 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:24.512627    2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-swqvw" podUID="5d6ec269-6c2a-4fe8-9e12-5a4315a9535e" containerName="manager" containerID="cri-o://e8480d607c68aa1ffd0f1876ac96fa3044a04fa9095cf111b98d7bb10150eee8" gracePeriod=10
Jul 07 15:52:24.753073 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:24.753049    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-swqvw"
Jul 07 15:52:24.825706 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:24.825665    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5d6ec269-6c2a-4fe8-9e12-5a4315a9535e-extensions-socket-volume\") pod \"5d6ec269-6c2a-4fe8-9e12-5a4315a9535e\" (UID: \"5d6ec269-6c2a-4fe8-9e12-5a4315a9535e\") "
Jul 07 15:52:24.825881 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:24.825770    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j6qwn\" (UniqueName: \"kubernetes.io/projected/5d6ec269-6c2a-4fe8-9e12-5a4315a9535e-kube-api-access-j6qwn\") pod \"5d6ec269-6c2a-4fe8-9e12-5a4315a9535e\" (UID: \"5d6ec269-6c2a-4fe8-9e12-5a4315a9535e\") "
Jul 07 15:52:24.826083 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:24.826059    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d6ec269-6c2a-4fe8-9e12-5a4315a9535e-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "5d6ec269-6c2a-4fe8-9e12-5a4315a9535e" (UID: "5d6ec269-6c2a-4fe8-9e12-5a4315a9535e"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 15:52:24.827776 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:24.827756    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d6ec269-6c2a-4fe8-9e12-5a4315a9535e-kube-api-access-j6qwn" (OuterVolumeSpecName: "kube-api-access-j6qwn") pod "5d6ec269-6c2a-4fe8-9e12-5a4315a9535e" (UID: "5d6ec269-6c2a-4fe8-9e12-5a4315a9535e"). InnerVolumeSpecName "kube-api-access-j6qwn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 15:52:24.927248 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:24.927159    2566 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5d6ec269-6c2a-4fe8-9e12-5a4315a9535e-extensions-socket-volume\") on node \"ip-10-0-138-101.ec2.internal\" DevicePath \"\""
Jul 07 15:52:24.927248 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:24.927188    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6qwn\" (UniqueName: \"kubernetes.io/projected/5d6ec269-6c2a-4fe8-9e12-5a4315a9535e-kube-api-access-j6qwn\") on node \"ip-10-0-138-101.ec2.internal\" DevicePath \"\""
Jul 07 15:52:25.447684 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:25.447648    2566 generic.go:356] "Generic (PLEG): container finished" podID="5d6ec269-6c2a-4fe8-9e12-5a4315a9535e" containerID="e8480d607c68aa1ffd0f1876ac96fa3044a04fa9095cf111b98d7bb10150eee8" exitCode=0
Jul 07 15:52:25.448108 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:25.447691    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-swqvw"
Jul 07 15:52:25.448108 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:25.447721    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-swqvw" event={"ID":"5d6ec269-6c2a-4fe8-9e12-5a4315a9535e","Type":"ContainerDied","Data":"e8480d607c68aa1ffd0f1876ac96fa3044a04fa9095cf111b98d7bb10150eee8"}
Jul 07 15:52:25.448108 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:25.447771    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-swqvw" event={"ID":"5d6ec269-6c2a-4fe8-9e12-5a4315a9535e","Type":"ContainerDied","Data":"e0bc608bf8630b40dbe9ea8de44484e2195778f5ed3012dd947107ff7b49e192"}
Jul 07 15:52:25.448108 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:25.447800    2566 scope.go:117] "RemoveContainer" containerID="e8480d607c68aa1ffd0f1876ac96fa3044a04fa9095cf111b98d7bb10150eee8"
Jul 07 15:52:25.456548 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:25.456531    2566 scope.go:117] "RemoveContainer" containerID="e8480d607c68aa1ffd0f1876ac96fa3044a04fa9095cf111b98d7bb10150eee8"
Jul 07 15:52:25.456825 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:52:25.456803    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8480d607c68aa1ffd0f1876ac96fa3044a04fa9095cf111b98d7bb10150eee8\": container with ID starting with e8480d607c68aa1ffd0f1876ac96fa3044a04fa9095cf111b98d7bb10150eee8 not found: ID does not exist" containerID="e8480d607c68aa1ffd0f1876ac96fa3044a04fa9095cf111b98d7bb10150eee8"
Jul 07 15:52:25.456870 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:25.456834    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8480d607c68aa1ffd0f1876ac96fa3044a04fa9095cf111b98d7bb10150eee8"} err="failed to get container status \"e8480d607c68aa1ffd0f1876ac96fa3044a04fa9095cf111b98d7bb10150eee8\": rpc error: code = NotFound desc = could not find container \"e8480d607c68aa1ffd0f1876ac96fa3044a04fa9095cf111b98d7bb10150eee8\": container with ID starting with e8480d607c68aa1ffd0f1876ac96fa3044a04fa9095cf111b98d7bb10150eee8 not found: ID does not exist"
Jul 07 15:52:25.473293 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:25.473266    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-swqvw"]
Jul 07 15:52:25.479695 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:25.479501    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-swqvw"]
Jul 07 15:52:26.475354 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:26.475322    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d6ec269-6c2a-4fe8-9e12-5a4315a9535e" path="/var/lib/kubelet/pods/5d6ec269-6c2a-4fe8-9e12-5a4315a9535e/volumes"
Jul 07 15:52:28.891105 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:28.891076    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nxk9c"]
Jul 07 15:52:28.891581 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:28.891419    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d6ec269-6c2a-4fe8-9e12-5a4315a9535e" containerName="manager"
Jul 07 15:52:28.891581 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:28.891434    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d6ec269-6c2a-4fe8-9e12-5a4315a9535e" containerName="manager"
Jul 07 15:52:28.891581 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:28.891512    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d6ec269-6c2a-4fe8-9e12-5a4315a9535e" containerName="manager"
Jul 07 15:52:28.894155 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:28.894134    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nxk9c"
Jul 07 15:52:28.896936 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:28.896914    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-7f79f\""
Jul 07 15:52:28.897409 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:28.897390    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 07 15:52:28.912668 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:28.912636    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nxk9c"]
Jul 07 15:52:28.958021 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:28.957980    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9ckh\" (UniqueName: \"kubernetes.io/projected/fc6066ec-a510-4f03-b98d-14c596d5b6ba-kube-api-access-d9ckh\") pod \"maas-default-gateway-openshift-default-5896d4dd49-nxk9c\" (UID: \"fc6066ec-a510-4f03-b98d-14c596d5b6ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nxk9c"
Jul 07 15:52:28.958021 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:28.958021    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc6066ec-a510-4f03-b98d-14c596d5b6ba-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-nxk9c\" (UID: \"fc6066ec-a510-4f03-b98d-14c596d5b6ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nxk9c"
Jul 07 15:52:28.958267 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:28.958084    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc6066ec-a510-4f03-b98d-14c596d5b6ba-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-nxk9c\" (UID: \"fc6066ec-a510-4f03-b98d-14c596d5b6ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nxk9c"
Jul 07 15:52:28.958267 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:28.958116    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc6066ec-a510-4f03-b98d-14c596d5b6ba-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-nxk9c\" (UID: \"fc6066ec-a510-4f03-b98d-14c596d5b6ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nxk9c"
Jul 07 15:52:28.958267 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:28.958182    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc6066ec-a510-4f03-b98d-14c596d5b6ba-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-nxk9c\" (UID: \"fc6066ec-a510-4f03-b98d-14c596d5b6ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nxk9c"
Jul 07 15:52:28.958267 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:28.958219    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc6066ec-a510-4f03-b98d-14c596d5b6ba-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-nxk9c\" (UID: \"fc6066ec-a510-4f03-b98d-14c596d5b6ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nxk9c"
Jul 07 15:52:28.958267 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:28.958249    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc6066ec-a510-4f03-b98d-14c596d5b6ba-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-nxk9c\" (UID: \"fc6066ec-a510-4f03-b98d-14c596d5b6ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nxk9c"
Jul 07 15:52:28.958480 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:28.958292    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc6066ec-a510-4f03-b98d-14c596d5b6ba-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-nxk9c\" (UID: \"fc6066ec-a510-4f03-b98d-14c596d5b6ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nxk9c"
Jul 07 15:52:28.958480 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:28.958327    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc6066ec-a510-4f03-b98d-14c596d5b6ba-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-nxk9c\" (UID: \"fc6066ec-a510-4f03-b98d-14c596d5b6ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nxk9c"
Jul 07 15:52:29.058871 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:29.058837    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9ckh\" (UniqueName: \"kubernetes.io/projected/fc6066ec-a510-4f03-b98d-14c596d5b6ba-kube-api-access-d9ckh\") pod \"maas-default-gateway-openshift-default-5896d4dd49-nxk9c\" (UID: \"fc6066ec-a510-4f03-b98d-14c596d5b6ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nxk9c"
Jul 07 15:52:29.059056 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:29.058879    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc6066ec-a510-4f03-b98d-14c596d5b6ba-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-nxk9c\" (UID: \"fc6066ec-a510-4f03-b98d-14c596d5b6ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nxk9c"
Jul 07 15:52:29.059056 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:29.058933    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc6066ec-a510-4f03-b98d-14c596d5b6ba-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-nxk9c\" (UID: \"fc6066ec-a510-4f03-b98d-14c596d5b6ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nxk9c"
Jul 07 15:52:29.059056 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:29.058955    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc6066ec-a510-4f03-b98d-14c596d5b6ba-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-nxk9c\" (UID: \"fc6066ec-a510-4f03-b98d-14c596d5b6ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nxk9c"
Jul 07 15:52:29.059056 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:29.058978    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc6066ec-a510-4f03-b98d-14c596d5b6ba-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-nxk9c\" (UID: \"fc6066ec-a510-4f03-b98d-14c596d5b6ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nxk9c"
Jul 07 15:52:29.059056 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:29.059001    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc6066ec-a510-4f03-b98d-14c596d5b6ba-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-nxk9c\" (UID: \"fc6066ec-a510-4f03-b98d-14c596d5b6ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nxk9c"
Jul 07 15:52:29.059056 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:29.059021    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc6066ec-a510-4f03-b98d-14c596d5b6ba-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-nxk9c\" (UID: \"fc6066ec-a510-4f03-b98d-14c596d5b6ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nxk9c"
Jul 07 15:52:29.059340 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:29.059058    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc6066ec-a510-4f03-b98d-14c596d5b6ba-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-nxk9c\" (UID: \"fc6066ec-a510-4f03-b98d-14c596d5b6ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nxk9c"
Jul 07 15:52:29.059340 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:29.059121    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc6066ec-a510-4f03-b98d-14c596d5b6ba-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-nxk9c\" (UID: \"fc6066ec-a510-4f03-b98d-14c596d5b6ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nxk9c"
Jul 07 15:52:29.059549 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:29.059504    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc6066ec-a510-4f03-b98d-14c596d5b6ba-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-nxk9c\" (UID: \"fc6066ec-a510-4f03-b98d-14c596d5b6ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nxk9c"
Jul 07 15:52:29.059643 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:29.059546    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc6066ec-a510-4f03-b98d-14c596d5b6ba-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-nxk9c\" (UID: \"fc6066ec-a510-4f03-b98d-14c596d5b6ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nxk9c"
Jul 07 15:52:29.059643 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:29.059593    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc6066ec-a510-4f03-b98d-14c596d5b6ba-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-nxk9c\" (UID: \"fc6066ec-a510-4f03-b98d-14c596d5b6ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nxk9c"
Jul 07 15:52:29.059789 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:29.059757    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc6066ec-a510-4f03-b98d-14c596d5b6ba-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-nxk9c\" (UID: \"fc6066ec-a510-4f03-b98d-14c596d5b6ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nxk9c"
Jul 07 15:52:29.060015 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:29.059998    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc6066ec-a510-4f03-b98d-14c596d5b6ba-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-nxk9c\" (UID: \"fc6066ec-a510-4f03-b98d-14c596d5b6ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nxk9c"
Jul 07 15:52:29.061952 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:29.061923    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc6066ec-a510-4f03-b98d-14c596d5b6ba-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-nxk9c\" (UID: \"fc6066ec-a510-4f03-b98d-14c596d5b6ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nxk9c"
Jul 07 15:52:29.062075 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:29.062059    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc6066ec-a510-4f03-b98d-14c596d5b6ba-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-nxk9c\" (UID: \"fc6066ec-a510-4f03-b98d-14c596d5b6ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nxk9c"
Jul 07 15:52:29.066675 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:29.066649    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc6066ec-a510-4f03-b98d-14c596d5b6ba-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-nxk9c\" (UID: \"fc6066ec-a510-4f03-b98d-14c596d5b6ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nxk9c"
Jul 07 15:52:29.066775 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:29.066756    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9ckh\" (UniqueName: \"kubernetes.io/projected/fc6066ec-a510-4f03-b98d-14c596d5b6ba-kube-api-access-d9ckh\") pod \"maas-default-gateway-openshift-default-5896d4dd49-nxk9c\" (UID: \"fc6066ec-a510-4f03-b98d-14c596d5b6ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nxk9c"
Jul 07 15:52:29.204946 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:29.204908    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nxk9c"
Jul 07 15:52:29.341924 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:29.341877    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nxk9c"]
Jul 07 15:52:29.344333 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:52:29.344310    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc6066ec_a510_4f03_b98d_14c596d5b6ba.slice/crio-3aebf0205878b1337f1e9cf0a4a312cd136e49aa4c17f612f69878b837c401ba WatchSource:0}: Error finding container 3aebf0205878b1337f1e9cf0a4a312cd136e49aa4c17f612f69878b837c401ba: Status 404 returned error can't find the container with id 3aebf0205878b1337f1e9cf0a4a312cd136e49aa4c17f612f69878b837c401ba
Jul 07 15:52:29.466478 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:29.466399    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nxk9c" event={"ID":"fc6066ec-a510-4f03-b98d-14c596d5b6ba","Type":"ContainerStarted","Data":"3aebf0205878b1337f1e9cf0a4a312cd136e49aa4c17f612f69878b837c401ba"}
Jul 07 15:52:31.878966 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:31.878929    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 07 15:52:31.879271 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:31.879002    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 07 15:52:31.879271 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:31.879036    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 07 15:52:32.484754 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:32.484710    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nxk9c" event={"ID":"fc6066ec-a510-4f03-b98d-14c596d5b6ba","Type":"ContainerStarted","Data":"12ca781abd7d4f3ad12ff2e5e6b600202dbfbf88145a8e085de1b7c45fe58711"}
Jul 07 15:52:32.509199 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:32.509150    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nxk9c" podStartSLOduration=1.976823024 podStartE2EDuration="4.509129336s" totalImagesPullingTime="2.532306312s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:52:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:52:32.508384327 +0000 UTC m=+506.596809998" watchObservedRunningTime="2026-07-07 15:52:32.509129336 +0000 UTC m=+506.597555008"
Jul 07 15:52:33.205975 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:33.205935    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nxk9c"
Jul 07 15:52:33.210646 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:33.210619    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nxk9c"
Jul 07 15:52:33.489277 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:33.489195    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nxk9c"
Jul 07 15:52:33.490261 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:33.490239    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nxk9c"
Jul 07 15:52:44.960910 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:44.958839    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mk99s"]
Jul 07 15:52:44.964419 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:44.964391    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-mk99s"
Jul 07 15:52:44.967369 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:44.967350    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-hmlvg\""
Jul 07 15:52:44.967369 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:44.967360    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 07 15:52:44.976470 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:44.976444    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mk99s"]
Jul 07 15:52:45.038872 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:45.038841    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mk99s"]
Jul 07 15:52:45.094250 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:45.094215    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pb74w\" (UniqueName: \"kubernetes.io/projected/c1c7a85b-891b-46ea-a556-8ab9e4f128f6-kube-api-access-pb74w\") pod \"limitador-limitador-54d6d988d5-mk99s\" (UID: \"c1c7a85b-891b-46ea-a556-8ab9e4f128f6\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-mk99s"
Jul 07 15:52:45.094250 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:45.094252    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c1c7a85b-891b-46ea-a556-8ab9e4f128f6-config-file\") pod \"limitador-limitador-54d6d988d5-mk99s\" (UID: \"c1c7a85b-891b-46ea-a556-8ab9e4f128f6\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-mk99s"
Jul 07 15:52:45.195531 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:45.195489    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pb74w\" (UniqueName: \"kubernetes.io/projected/c1c7a85b-891b-46ea-a556-8ab9e4f128f6-kube-api-access-pb74w\") pod \"limitador-limitador-54d6d988d5-mk99s\" (UID: \"c1c7a85b-891b-46ea-a556-8ab9e4f128f6\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-mk99s"
Jul 07 15:52:45.195694 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:45.195538    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c1c7a85b-891b-46ea-a556-8ab9e4f128f6-config-file\") pod \"limitador-limitador-54d6d988d5-mk99s\" (UID: \"c1c7a85b-891b-46ea-a556-8ab9e4f128f6\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-mk99s"
Jul 07 15:52:45.196425 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:45.196405    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c1c7a85b-891b-46ea-a556-8ab9e4f128f6-config-file\") pod \"limitador-limitador-54d6d988d5-mk99s\" (UID: \"c1c7a85b-891b-46ea-a556-8ab9e4f128f6\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-mk99s"
Jul 07 15:52:45.203624 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:45.203595    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pb74w\" (UniqueName: \"kubernetes.io/projected/c1c7a85b-891b-46ea-a556-8ab9e4f128f6-kube-api-access-pb74w\") pod \"limitador-limitador-54d6d988d5-mk99s\" (UID: \"c1c7a85b-891b-46ea-a556-8ab9e4f128f6\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-mk99s"
Jul 07 15:52:45.274647 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:45.274566    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-mk99s"
Jul 07 15:52:45.406286 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:45.406251    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mk99s"]
Jul 07 15:52:45.409566 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:52:45.409539    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1c7a85b_891b_46ea_a556_8ab9e4f128f6.slice/crio-f9c862a7a42bee8c4e29caccbd6e419da26259db6aa89eee47ddc5448b7c11da WatchSource:0}: Error finding container f9c862a7a42bee8c4e29caccbd6e419da26259db6aa89eee47ddc5448b7c11da: Status 404 returned error can't find the container with id f9c862a7a42bee8c4e29caccbd6e419da26259db6aa89eee47ddc5448b7c11da
Jul 07 15:52:45.427595 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:45.427566    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-wx7gf"]
Jul 07 15:52:45.432453 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:45.432433    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-wx7gf"
Jul 07 15:52:45.434641 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:45.434620    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-mj9p5\""
Jul 07 15:52:45.437975 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:45.437952    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-wx7gf"]
Jul 07 15:52:45.498091 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:45.498064    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpx2j\" (UniqueName: \"kubernetes.io/projected/c60393ef-bea0-40b9-8da3-5e5d56a1e445-kube-api-access-qpx2j\") pod \"authorino-58cd955675-wx7gf\" (UID: \"c60393ef-bea0-40b9-8da3-5e5d56a1e445\") " pod="kuadrant-system/authorino-58cd955675-wx7gf"
Jul 07 15:52:45.540947 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:45.540848    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-mk99s" event={"ID":"c1c7a85b-891b-46ea-a556-8ab9e4f128f6","Type":"ContainerStarted","Data":"f9c862a7a42bee8c4e29caccbd6e419da26259db6aa89eee47ddc5448b7c11da"}
Jul 07 15:52:45.599027 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:45.598992    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpx2j\" (UniqueName: \"kubernetes.io/projected/c60393ef-bea0-40b9-8da3-5e5d56a1e445-kube-api-access-qpx2j\") pod \"authorino-58cd955675-wx7gf\" (UID: \"c60393ef-bea0-40b9-8da3-5e5d56a1e445\") " pod="kuadrant-system/authorino-58cd955675-wx7gf"
Jul 07 15:52:45.607443 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:45.607413    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpx2j\" (UniqueName: \"kubernetes.io/projected/c60393ef-bea0-40b9-8da3-5e5d56a1e445-kube-api-access-qpx2j\") pod \"authorino-58cd955675-wx7gf\" (UID: \"c60393ef-bea0-40b9-8da3-5e5d56a1e445\") " pod="kuadrant-system/authorino-58cd955675-wx7gf"
Jul 07 15:52:45.743767 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:45.743734    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-wx7gf"
Jul 07 15:52:45.871757 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:45.871724    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-wx7gf"]
Jul 07 15:52:45.874793 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:52:45.874764    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc60393ef_bea0_40b9_8da3_5e5d56a1e445.slice/crio-3538235a3be117b93430f8c808cf79b3ef6d04a8b97ff28ceb28d92a66cef8df WatchSource:0}: Error finding container 3538235a3be117b93430f8c808cf79b3ef6d04a8b97ff28ceb28d92a66cef8df: Status 404 returned error can't find the container with id 3538235a3be117b93430f8c808cf79b3ef6d04a8b97ff28ceb28d92a66cef8df
Jul 07 15:52:46.554136 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:46.554054    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-wx7gf" event={"ID":"c60393ef-bea0-40b9-8da3-5e5d56a1e445","Type":"ContainerStarted","Data":"3538235a3be117b93430f8c808cf79b3ef6d04a8b97ff28ceb28d92a66cef8df"}
Jul 07 15:52:50.523270 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:50.523235    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-wx7gf"]
Jul 07 15:52:50.574708 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:50.574673    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-mk99s" event={"ID":"c1c7a85b-891b-46ea-a556-8ab9e4f128f6","Type":"ContainerStarted","Data":"c16a3efe435d231346083ba81e0de7695382edda12ac0fdd8cd5ce75b148ae5c"}
Jul 07 15:52:50.574925 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:50.574866    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-mk99s"
Jul 07 15:52:50.576257 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:50.576221    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-wx7gf" event={"ID":"c60393ef-bea0-40b9-8da3-5e5d56a1e445","Type":"ContainerStarted","Data":"eb8f69f6cb6a021c13b18a3879beacaa1f8380f7db6dbcef0c962e48ef5c2b82"}
Jul 07 15:52:50.591750 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:50.591701    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-mk99s" podStartSLOduration=2.112631035 podStartE2EDuration="6.591686461s" totalImagesPullingTime="4.479055426s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:52:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:52:50.590413736 +0000 UTC m=+524.678839407" watchObservedRunningTime="2026-07-07 15:52:50.591686461 +0000 UTC m=+524.680112131"
Jul 07 15:52:50.604826 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:50.604781    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-wx7gf" podStartSLOduration=1.58975915 podStartE2EDuration="5.604767869s" totalImagesPullingTime="4.015008719s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:52:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:52:50.604258846 +0000 UTC m=+524.692684516" watchObservedRunningTime="2026-07-07 15:52:50.604767869 +0000 UTC m=+524.693193539"
Jul 07 15:52:51.580556 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:51.580514    2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-wx7gf" podUID="c60393ef-bea0-40b9-8da3-5e5d56a1e445" containerName="authorino" containerID="cri-o://eb8f69f6cb6a021c13b18a3879beacaa1f8380f7db6dbcef0c962e48ef5c2b82" gracePeriod=30
Jul 07 15:52:51.839428 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:51.839369    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-wx7gf"
Jul 07 15:52:51.957684 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:51.957654    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qpx2j\" (UniqueName: \"kubernetes.io/projected/c60393ef-bea0-40b9-8da3-5e5d56a1e445-kube-api-access-qpx2j\") pod \"c60393ef-bea0-40b9-8da3-5e5d56a1e445\" (UID: \"c60393ef-bea0-40b9-8da3-5e5d56a1e445\") "
Jul 07 15:52:51.959741 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:51.959714    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c60393ef-bea0-40b9-8da3-5e5d56a1e445-kube-api-access-qpx2j" (OuterVolumeSpecName: "kube-api-access-qpx2j") pod "c60393ef-bea0-40b9-8da3-5e5d56a1e445" (UID: "c60393ef-bea0-40b9-8da3-5e5d56a1e445"). InnerVolumeSpecName "kube-api-access-qpx2j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 15:52:52.058273 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:52.058225    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpx2j\" (UniqueName: \"kubernetes.io/projected/c60393ef-bea0-40b9-8da3-5e5d56a1e445-kube-api-access-qpx2j\") on node \"ip-10-0-138-101.ec2.internal\" DevicePath \"\""
Jul 07 15:52:52.585579 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:52.585548    2566 generic.go:356] "Generic (PLEG): container finished" podID="c60393ef-bea0-40b9-8da3-5e5d56a1e445" containerID="eb8f69f6cb6a021c13b18a3879beacaa1f8380f7db6dbcef0c962e48ef5c2b82" exitCode=0
Jul 07 15:52:52.586033 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:52.585590    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-wx7gf"
Jul 07 15:52:52.586033 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:52.585607    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-wx7gf" event={"ID":"c60393ef-bea0-40b9-8da3-5e5d56a1e445","Type":"ContainerDied","Data":"eb8f69f6cb6a021c13b18a3879beacaa1f8380f7db6dbcef0c962e48ef5c2b82"}
Jul 07 15:52:52.586033 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:52.585637    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-wx7gf" event={"ID":"c60393ef-bea0-40b9-8da3-5e5d56a1e445","Type":"ContainerDied","Data":"3538235a3be117b93430f8c808cf79b3ef6d04a8b97ff28ceb28d92a66cef8df"}
Jul 07 15:52:52.586033 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:52.585653    2566 scope.go:117] "RemoveContainer" containerID="eb8f69f6cb6a021c13b18a3879beacaa1f8380f7db6dbcef0c962e48ef5c2b82"
Jul 07 15:52:52.594168 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:52.594145    2566 scope.go:117] "RemoveContainer" containerID="eb8f69f6cb6a021c13b18a3879beacaa1f8380f7db6dbcef0c962e48ef5c2b82"
Jul 07 15:52:52.594434 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:52:52.594417    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb8f69f6cb6a021c13b18a3879beacaa1f8380f7db6dbcef0c962e48ef5c2b82\": container with ID starting with eb8f69f6cb6a021c13b18a3879beacaa1f8380f7db6dbcef0c962e48ef5c2b82 not found: ID does not exist" containerID="eb8f69f6cb6a021c13b18a3879beacaa1f8380f7db6dbcef0c962e48ef5c2b82"
Jul 07 15:52:52.594482 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:52.594444    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb8f69f6cb6a021c13b18a3879beacaa1f8380f7db6dbcef0c962e48ef5c2b82"} err="failed to get container status \"eb8f69f6cb6a021c13b18a3879beacaa1f8380f7db6dbcef0c962e48ef5c2b82\": rpc error: code = NotFound desc = could not find container \"eb8f69f6cb6a021c13b18a3879beacaa1f8380f7db6dbcef0c962e48ef5c2b82\": container with ID starting with eb8f69f6cb6a021c13b18a3879beacaa1f8380f7db6dbcef0c962e48ef5c2b82 not found: ID does not exist"
Jul 07 15:52:52.602953 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:52.602930    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-wx7gf"]
Jul 07 15:52:52.606587 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:52.606567    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-wx7gf"]
Jul 07 15:52:54.474622 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:52:54.474589    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c60393ef-bea0-40b9-8da3-5e5d56a1e445" path="/var/lib/kubelet/pods/c60393ef-bea0-40b9-8da3-5e5d56a1e445/volumes"
Jul 07 15:53:01.581912 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:53:01.581808    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-mk99s"
Jul 07 15:53:18.385114 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:53:18.385079    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-76877994-8gz6n"]
Jul 07 15:53:18.385581 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:53:18.385401    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c60393ef-bea0-40b9-8da3-5e5d56a1e445" containerName="authorino"
Jul 07 15:53:18.385581 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:53:18.385412    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c60393ef-bea0-40b9-8da3-5e5d56a1e445" containerName="authorino"
Jul 07 15:53:18.385581 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:53:18.385475    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="c60393ef-bea0-40b9-8da3-5e5d56a1e445" containerName="authorino"
Jul 07 15:53:18.392350 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:53:18.392323    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-8gz6n"
Jul 07 15:53:18.395387 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:53:18.395363    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-mj9p5\""
Jul 07 15:53:18.395521 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:53:18.395367    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 07 15:53:18.397481 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:53:18.397453    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-76877994-8gz6n"]
Jul 07 15:53:18.419658 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:53:18.419628    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76877994-8gz6n"]
Jul 07 15:53:18.419950 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:53:18.419930    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-gd6gv tls-cert], unattached volumes=[], failed to process volumes=[kube-api-access-gd6gv tls-cert]: context canceled" pod="kuadrant-system/authorino-76877994-8gz6n" podUID="8700a58d-fbd7-4c94-a924-e6af5292c7e2"
Jul 07 15:53:18.449771 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:53:18.449734    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7f5d749cb4-qncrn"]
Jul 07 15:53:18.453134 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:53:18.453113    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7f5d749cb4-qncrn"
Jul 07 15:53:18.460859 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:53:18.460832    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7f5d749cb4-qncrn"]
Jul 07 15:53:18.467419 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:53:18.467395    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfjht\" (UniqueName: \"kubernetes.io/projected/ae1ed14a-cd55-40b6-a8e1-b567ef0dd040-kube-api-access-vfjht\") pod \"authorino-7f5d749cb4-qncrn\" (UID: \"ae1ed14a-cd55-40b6-a8e1-b567ef0dd040\") " pod="kuadrant-system/authorino-7f5d749cb4-qncrn"
Jul 07 15:53:18.467540 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:53:18.467427    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ae1ed14a-cd55-40b6-a8e1-b567ef0dd040-tls-cert\") pod \"authorino-7f5d749cb4-qncrn\" (UID: \"ae1ed14a-cd55-40b6-a8e1-b567ef0dd040\") " pod="kuadrant-system/authorino-7f5d749cb4-qncrn"
Jul 07 15:53:18.567910 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:53:18.567842    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gd6gv\" (UniqueName: \"kubernetes.io/projected/8700a58d-fbd7-4c94-a924-e6af5292c7e2-kube-api-access-gd6gv\") pod \"authorino-76877994-8gz6n\" (UID: \"8700a58d-fbd7-4c94-a924-e6af5292c7e2\") " pod="kuadrant-system/authorino-76877994-8gz6n"
Jul 07 15:53:18.568089 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:53:18.567958    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8700a58d-fbd7-4c94-a924-e6af5292c7e2-tls-cert\") pod \"authorino-76877994-8gz6n\" (UID: \"8700a58d-fbd7-4c94-a924-e6af5292c7e2\") " pod="kuadrant-system/authorino-76877994-8gz6n"
Jul 07 15:53:18.568089 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:53:18.567990    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfjht\" (UniqueName: \"kubernetes.io/projected/ae1ed14a-cd55-40b6-a8e1-b567ef0dd040-kube-api-access-vfjht\") pod \"authorino-7f5d749cb4-qncrn\" (UID: \"ae1ed14a-cd55-40b6-a8e1-b567ef0dd040\") " pod="kuadrant-system/authorino-7f5d749cb4-qncrn"
Jul 07 15:53:18.568089 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:53:18.568014    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ae1ed14a-cd55-40b6-a8e1-b567ef0dd040-tls-cert\") pod \"authorino-7f5d749cb4-qncrn\" (UID: \"ae1ed14a-cd55-40b6-a8e1-b567ef0dd040\") " pod="kuadrant-system/authorino-7f5d749cb4-qncrn"
Jul 07 15:53:18.570394 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:53:18.570369    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ae1ed14a-cd55-40b6-a8e1-b567ef0dd040-tls-cert\") pod \"authorino-7f5d749cb4-qncrn\" (UID: \"ae1ed14a-cd55-40b6-a8e1-b567ef0dd040\") " pod="kuadrant-system/authorino-7f5d749cb4-qncrn"
Jul 07 15:53:18.578636 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:53:18.578615    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfjht\" (UniqueName: \"kubernetes.io/projected/ae1ed14a-cd55-40b6-a8e1-b567ef0dd040-kube-api-access-vfjht\") pod \"authorino-7f5d749cb4-qncrn\" (UID: \"ae1ed14a-cd55-40b6-a8e1-b567ef0dd040\") " pod="kuadrant-system/authorino-7f5d749cb4-qncrn"
Jul 07 15:53:18.668825 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:53:18.668738    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd6gv\" (UniqueName: \"kubernetes.io/projected/8700a58d-fbd7-4c94-a924-e6af5292c7e2-kube-api-access-gd6gv\") pod \"authorino-76877994-8gz6n\" (UID: \"8700a58d-fbd7-4c94-a924-e6af5292c7e2\") " pod="kuadrant-system/authorino-76877994-8gz6n"
Jul 07 15:53:18.668825 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:53:18.668823    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8700a58d-fbd7-4c94-a924-e6af5292c7e2-tls-cert\") pod \"authorino-76877994-8gz6n\" (UID: \"8700a58d-fbd7-4c94-a924-e6af5292c7e2\") " pod="kuadrant-system/authorino-76877994-8gz6n"
Jul 07 15:53:18.671297 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:53:18.671267    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8700a58d-fbd7-4c94-a924-e6af5292c7e2-tls-cert\") pod \"authorino-76877994-8gz6n\" (UID: \"8700a58d-fbd7-4c94-a924-e6af5292c7e2\") " pod="kuadrant-system/authorino-76877994-8gz6n"
Jul 07 15:53:18.677872 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:53:18.677844    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gd6gv\" (UniqueName: \"kubernetes.io/projected/8700a58d-fbd7-4c94-a924-e6af5292c7e2-kube-api-access-gd6gv\") pod \"authorino-76877994-8gz6n\" (UID: \"8700a58d-fbd7-4c94-a924-e6af5292c7e2\") " pod="kuadrant-system/authorino-76877994-8gz6n"
Jul 07 15:53:18.699551 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:53:18.699521    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-8gz6n"
Jul 07 15:53:18.707042 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:53:18.707018    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-8gz6n"
Jul 07 15:53:18.762855 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:53:18.762813    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7f5d749cb4-qncrn"
Jul 07 15:53:18.870432 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:53:18.870399    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gd6gv\" (UniqueName: \"kubernetes.io/projected/8700a58d-fbd7-4c94-a924-e6af5292c7e2-kube-api-access-gd6gv\") pod \"8700a58d-fbd7-4c94-a924-e6af5292c7e2\" (UID: \"8700a58d-fbd7-4c94-a924-e6af5292c7e2\") "
Jul 07 15:53:18.870595 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:53:18.870445    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8700a58d-fbd7-4c94-a924-e6af5292c7e2-tls-cert\") pod \"8700a58d-fbd7-4c94-a924-e6af5292c7e2\" (UID: \"8700a58d-fbd7-4c94-a924-e6af5292c7e2\") "
Jul 07 15:53:18.873192 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:53:18.873160    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8700a58d-fbd7-4c94-a924-e6af5292c7e2-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "8700a58d-fbd7-4c94-a924-e6af5292c7e2" (UID: "8700a58d-fbd7-4c94-a924-e6af5292c7e2"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 15:53:18.873336 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:53:18.873195    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8700a58d-fbd7-4c94-a924-e6af5292c7e2-kube-api-access-gd6gv" (OuterVolumeSpecName: "kube-api-access-gd6gv") pod "8700a58d-fbd7-4c94-a924-e6af5292c7e2" (UID: "8700a58d-fbd7-4c94-a924-e6af5292c7e2"). InnerVolumeSpecName "kube-api-access-gd6gv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 15:53:18.914473 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:53:18.914337    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7f5d749cb4-qncrn"]
Jul 07 15:53:18.917194 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:53:18.917160    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podae1ed14a_cd55_40b6_a8e1_b567ef0dd040.slice/crio-411175616b55e11a6839e5610dc9f4cf7ce9563001ebcf863eb6745fb780fcab WatchSource:0}: Error finding container 411175616b55e11a6839e5610dc9f4cf7ce9563001ebcf863eb6745fb780fcab: Status 404 returned error can't find the container with id 411175616b55e11a6839e5610dc9f4cf7ce9563001ebcf863eb6745fb780fcab
Jul 07 15:53:18.971114 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:53:18.971076    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gd6gv\" (UniqueName: \"kubernetes.io/projected/8700a58d-fbd7-4c94-a924-e6af5292c7e2-kube-api-access-gd6gv\") on node \"ip-10-0-138-101.ec2.internal\" DevicePath \"\""
Jul 07 15:53:18.971114 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:53:18.971103    2566 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8700a58d-fbd7-4c94-a924-e6af5292c7e2-tls-cert\") on node \"ip-10-0-138-101.ec2.internal\" DevicePath \"\""
Jul 07 15:53:19.705663 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:53:19.705621    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7f5d749cb4-qncrn" event={"ID":"ae1ed14a-cd55-40b6-a8e1-b567ef0dd040","Type":"ContainerStarted","Data":"c64c9f498ef3c8187660aa6a88b9bc21a6158bf8f09fffd8e080007de085705c"}
Jul 07 15:53:19.706088 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:53:19.705677    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7f5d749cb4-qncrn" event={"ID":"ae1ed14a-cd55-40b6-a8e1-b567ef0dd040","Type":"ContainerStarted","Data":"411175616b55e11a6839e5610dc9f4cf7ce9563001ebcf863eb6745fb780fcab"}
Jul 07 15:53:19.706088 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:53:19.705634    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-8gz6n"
Jul 07 15:53:19.725266 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:53:19.725205    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7f5d749cb4-qncrn" podStartSLOduration=1.341559685 podStartE2EDuration="1.725189157s" totalImagesPullingTime="383.629472ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:53:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:53:19.723809171 +0000 UTC m=+553.812234841" watchObservedRunningTime="2026-07-07 15:53:19.725189157 +0000 UTC m=+553.813614827"
Jul 07 15:53:19.749385 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:53:19.749343    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76877994-8gz6n"]
Jul 07 15:53:19.756605 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:53:19.756571    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-76877994-8gz6n"]
Jul 07 15:53:20.474519 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:53:20.474490    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8700a58d-fbd7-4c94-a924-e6af5292c7e2" path="/var/lib/kubelet/pods/8700a58d-fbd7-4c94-a924-e6af5292c7e2/volumes"
Jul 07 15:54:01.658464 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:54:01.658421    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mk99s"]
Jul 07 15:54:09.924269 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:54:09.924233    2566 generic.go:356] "Generic (PLEG): container finished" podID="fc6066ec-a510-4f03-b98d-14c596d5b6ba" containerID="12ca781abd7d4f3ad12ff2e5e6b600202dbfbf88145a8e085de1b7c45fe58711" exitCode=0
Jul 07 15:54:09.924747 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:54:09.924308    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nxk9c" event={"ID":"fc6066ec-a510-4f03-b98d-14c596d5b6ba","Type":"ContainerDied","Data":"12ca781abd7d4f3ad12ff2e5e6b600202dbfbf88145a8e085de1b7c45fe58711"}
Jul 07 15:54:09.924747 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:54:09.924723    2566 scope.go:117] "RemoveContainer" containerID="12ca781abd7d4f3ad12ff2e5e6b600202dbfbf88145a8e085de1b7c45fe58711"
Jul 07 15:54:09.925716 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:54:09.925689    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 07 15:54:09.925764 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:54:09.925755    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 07 15:54:09.925797 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:54:09.925787    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 07 15:54:10.931115 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:54:10.931083    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nxk9c" event={"ID":"fc6066ec-a510-4f03-b98d-14c596d5b6ba","Type":"ContainerStarted","Data":"1f851b2ef6032ddbadaac0a1d641a329205a630c0801d6b93802be73dabdc65a"}
Jul 07 15:54:11.205675 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:54:11.205636    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nxk9c"
Jul 07 15:54:11.205971 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:54:11.205948    2566 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-nxk9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Jul 07 15:54:11.206025 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:54:11.206007    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nxk9c" podUID="fc6066ec-a510-4f03-b98d-14c596d5b6ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Jul 07 15:54:12.206239 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:54:12.206193    2566 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-nxk9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Jul 07 15:54:12.206641 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:54:12.206267    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nxk9c" podUID="fc6066ec-a510-4f03-b98d-14c596d5b6ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Jul 07 15:54:13.205338 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:54:13.205290    2566 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-nxk9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Jul 07 15:54:13.205510 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:54:13.205354    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nxk9c" podUID="fc6066ec-a510-4f03-b98d-14c596d5b6ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Jul 07 15:54:14.206335 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:54:14.206294    2566 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-nxk9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Jul 07 15:54:14.206706 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:54:14.206388    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nxk9c" podUID="fc6066ec-a510-4f03-b98d-14c596d5b6ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Jul 07 15:54:15.209527 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:54:15.209494    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nxk9c"
Jul 07 15:54:15.210007 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:54:15.209755    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nxk9c"
Jul 07 15:54:15.210637 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:54:15.210619    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nxk9c"
Jul 07 15:54:17.831644 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:54:17.831605    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mk99s"]
Jul 07 15:54:37.734464 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:54:37.734388    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mk99s"]
Jul 07 15:54:52.053345 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:54:52.053301    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mk99s"]
Jul 07 15:55:03.742827 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:55:03.742785    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mk99s"]
Jul 07 15:55:11.327312 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:55:11.327267    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mk99s"]
Jul 07 15:55:13.838578 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:55:13.838546    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-pc5ps"]
Jul 07 15:55:13.841809 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:55:13.841791    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-pc5ps"
Jul 07 15:55:13.846407 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:55:13.846380    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-vvllp\""
Jul 07 15:55:13.846407 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:55:13.846369    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 07 15:55:13.846593 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:55:13.846404    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Jul 07 15:55:13.846769 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:55:13.846753    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 07 15:55:13.855361 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:55:13.855334    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-pc5ps"]
Jul 07 15:55:13.867464 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:55:13.867437    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ccee4b1b-9e55-44b7-8374-5374d67a0fa3-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-pc5ps\" (UID: \"ccee4b1b-9e55-44b7-8374-5374d67a0fa3\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-pc5ps"
Jul 07 15:55:13.867603 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:55:13.867483    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ccee4b1b-9e55-44b7-8374-5374d67a0fa3-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-pc5ps\" (UID: \"ccee4b1b-9e55-44b7-8374-5374d67a0fa3\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-pc5ps"
Jul 07 15:55:13.867603 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:55:13.867503    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ccee4b1b-9e55-44b7-8374-5374d67a0fa3-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-pc5ps\" (UID: \"ccee4b1b-9e55-44b7-8374-5374d67a0fa3\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-pc5ps"
Jul 07 15:55:13.867603 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:55:13.867554    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgw2s\" (UniqueName: \"kubernetes.io/projected/ccee4b1b-9e55-44b7-8374-5374d67a0fa3-kube-api-access-rgw2s\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-pc5ps\" (UID: \"ccee4b1b-9e55-44b7-8374-5374d67a0fa3\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-pc5ps"
Jul 07 15:55:13.867756 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:55:13.867610    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ccee4b1b-9e55-44b7-8374-5374d67a0fa3-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-pc5ps\" (UID: \"ccee4b1b-9e55-44b7-8374-5374d67a0fa3\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-pc5ps"
Jul 07 15:55:13.867756 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:55:13.867658    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ccee4b1b-9e55-44b7-8374-5374d67a0fa3-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-pc5ps\" (UID: \"ccee4b1b-9e55-44b7-8374-5374d67a0fa3\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-pc5ps"
Jul 07 15:55:13.968576 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:55:13.968525    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ccee4b1b-9e55-44b7-8374-5374d67a0fa3-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-pc5ps\" (UID: \"ccee4b1b-9e55-44b7-8374-5374d67a0fa3\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-pc5ps"
Jul 07 15:55:13.968772 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:55:13.968702    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ccee4b1b-9e55-44b7-8374-5374d67a0fa3-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-pc5ps\" (UID: \"ccee4b1b-9e55-44b7-8374-5374d67a0fa3\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-pc5ps"
Jul 07 15:55:13.968772 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:55:13.968745    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ccee4b1b-9e55-44b7-8374-5374d67a0fa3-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-pc5ps\" (UID: \"ccee4b1b-9e55-44b7-8374-5374d67a0fa3\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-pc5ps"
Jul 07 15:55:13.968942 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:55:13.968921    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgw2s\" (UniqueName: \"kubernetes.io/projected/ccee4b1b-9e55-44b7-8374-5374d67a0fa3-kube-api-access-rgw2s\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-pc5ps\" (UID: \"ccee4b1b-9e55-44b7-8374-5374d67a0fa3\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-pc5ps"
Jul 07 15:55:13.969056 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:55:13.969036    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ccee4b1b-9e55-44b7-8374-5374d67a0fa3-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-pc5ps\" (UID: \"ccee4b1b-9e55-44b7-8374-5374d67a0fa3\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-pc5ps"
Jul 07 15:55:13.969146 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:55:13.969055    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ccee4b1b-9e55-44b7-8374-5374d67a0fa3-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-pc5ps\" (UID: \"ccee4b1b-9e55-44b7-8374-5374d67a0fa3\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-pc5ps"
Jul 07 15:55:13.969146 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:55:13.969138    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ccee4b1b-9e55-44b7-8374-5374d67a0fa3-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-pc5ps\" (UID: \"ccee4b1b-9e55-44b7-8374-5374d67a0fa3\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-pc5ps"
Jul 07 15:55:13.969256 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:55:13.969159    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ccee4b1b-9e55-44b7-8374-5374d67a0fa3-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-pc5ps\" (UID: \"ccee4b1b-9e55-44b7-8374-5374d67a0fa3\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-pc5ps"
Jul 07 15:55:13.969450 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:55:13.969428    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ccee4b1b-9e55-44b7-8374-5374d67a0fa3-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-pc5ps\" (UID: \"ccee4b1b-9e55-44b7-8374-5374d67a0fa3\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-pc5ps"
Jul 07 15:55:13.971105 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:55:13.971084    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ccee4b1b-9e55-44b7-8374-5374d67a0fa3-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-pc5ps\" (UID: \"ccee4b1b-9e55-44b7-8374-5374d67a0fa3\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-pc5ps"
Jul 07 15:55:13.971771 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:55:13.971749    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ccee4b1b-9e55-44b7-8374-5374d67a0fa3-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-pc5ps\" (UID: \"ccee4b1b-9e55-44b7-8374-5374d67a0fa3\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-pc5ps"
Jul 07 15:55:13.977710 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:55:13.977676    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgw2s\" (UniqueName: \"kubernetes.io/projected/ccee4b1b-9e55-44b7-8374-5374d67a0fa3-kube-api-access-rgw2s\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-pc5ps\" (UID: \"ccee4b1b-9e55-44b7-8374-5374d67a0fa3\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-pc5ps"
Jul 07 15:55:14.151094 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:55:14.151007    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-pc5ps"
Jul 07 15:55:14.295246 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:55:14.295219    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-pc5ps"]
Jul 07 15:55:14.298006 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:55:14.297978    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podccee4b1b_9e55_44b7_8374_5374d67a0fa3.slice/crio-b1ae1a46ae96b23db041dec2a06035534c77fb2c49ad5b49d1f477177c6b3055 WatchSource:0}: Error finding container b1ae1a46ae96b23db041dec2a06035534c77fb2c49ad5b49d1f477177c6b3055: Status 404 returned error can't find the container with id b1ae1a46ae96b23db041dec2a06035534c77fb2c49ad5b49d1f477177c6b3055
Jul 07 15:55:14.299434 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:55:14.299408    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 15:55:15.230500 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:55:15.230459    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-pc5ps" event={"ID":"ccee4b1b-9e55-44b7-8374-5374d67a0fa3","Type":"ContainerStarted","Data":"b1ae1a46ae96b23db041dec2a06035534c77fb2c49ad5b49d1f477177c6b3055"}
Jul 07 15:55:16.237233 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:55:16.237200    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-pc5ps" event={"ID":"ccee4b1b-9e55-44b7-8374-5374d67a0fa3","Type":"ContainerStarted","Data":"747fc4a182d9d738d43b6887edfc1dccb17a7b2f8539cec6d798d312376d49a0"}
Jul 07 15:55:16.258657 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:55:16.258606    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-pc5ps" podStartSLOduration=2.072574081 podStartE2EDuration="3.258590325s" totalImagesPullingTime="1.186016244s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:55:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:55:16.257005801 +0000 UTC m=+670.345431465" watchObservedRunningTime="2026-07-07 15:55:16.258590325 +0000 UTC m=+670.347015995"
Jul 07 15:55:18.334357 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:55:18.334322    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mk99s"]
Jul 07 15:55:24.151990 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:55:24.151953    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-pc5ps"
Jul 07 15:55:24.152452 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:55:24.152006    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-pc5ps"
Jul 07 15:55:24.164837 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:55:24.164812    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-pc5ps"
Jul 07 15:55:24.286988 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:55:24.286957    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-pc5ps"
Jul 07 15:56:35.727234 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:56:35.727199    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56b56968cc-gxml2"]
Jul 07 15:56:35.730295 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:56:35.730274    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56b56968cc-gxml2"
Jul 07 15:56:35.741525 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:56:35.741497    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56b56968cc-gxml2"]
Jul 07 15:56:35.893409 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:56:35.893367    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sljh7\" (UniqueName: \"kubernetes.io/projected/fde2c89a-f756-424d-9630-8993c5eee62c-kube-api-access-sljh7\") pod \"authorino-56b56968cc-gxml2\" (UID: \"fde2c89a-f756-424d-9630-8993c5eee62c\") " pod="kuadrant-system/authorino-56b56968cc-gxml2"
Jul 07 15:56:35.893592 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:56:35.893444    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fde2c89a-f756-424d-9630-8993c5eee62c-tls-cert\") pod \"authorino-56b56968cc-gxml2\" (UID: \"fde2c89a-f756-424d-9630-8993c5eee62c\") " pod="kuadrant-system/authorino-56b56968cc-gxml2"
Jul 07 15:56:35.994856 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:56:35.994779    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sljh7\" (UniqueName: \"kubernetes.io/projected/fde2c89a-f756-424d-9630-8993c5eee62c-kube-api-access-sljh7\") pod \"authorino-56b56968cc-gxml2\" (UID: \"fde2c89a-f756-424d-9630-8993c5eee62c\") " pod="kuadrant-system/authorino-56b56968cc-gxml2"
Jul 07 15:56:35.994856 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:56:35.994839    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fde2c89a-f756-424d-9630-8993c5eee62c-tls-cert\") pod \"authorino-56b56968cc-gxml2\" (UID: \"fde2c89a-f756-424d-9630-8993c5eee62c\") " pod="kuadrant-system/authorino-56b56968cc-gxml2"
Jul 07 15:56:35.997256 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:56:35.997228    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fde2c89a-f756-424d-9630-8993c5eee62c-tls-cert\") pod \"authorino-56b56968cc-gxml2\" (UID: \"fde2c89a-f756-424d-9630-8993c5eee62c\") " pod="kuadrant-system/authorino-56b56968cc-gxml2"
Jul 07 15:56:36.003920 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:56:36.003857    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sljh7\" (UniqueName: \"kubernetes.io/projected/fde2c89a-f756-424d-9630-8993c5eee62c-kube-api-access-sljh7\") pod \"authorino-56b56968cc-gxml2\" (UID: \"fde2c89a-f756-424d-9630-8993c5eee62c\") " pod="kuadrant-system/authorino-56b56968cc-gxml2"
Jul 07 15:56:36.039520 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:56:36.039483    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56b56968cc-gxml2"
Jul 07 15:56:36.185415 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:56:36.185277    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56b56968cc-gxml2"]
Jul 07 15:56:36.188134 ip-10-0-138-101 kubenswrapper[2566]: W0707 15:56:36.188098    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfde2c89a_f756_424d_9630_8993c5eee62c.slice/crio-38a4cac322de1b47f74c782e62dfae47eb200f3308633ca9d43fec2f4bf7b50c WatchSource:0}: Error finding container 38a4cac322de1b47f74c782e62dfae47eb200f3308633ca9d43fec2f4bf7b50c: Status 404 returned error can't find the container with id 38a4cac322de1b47f74c782e62dfae47eb200f3308633ca9d43fec2f4bf7b50c
Jul 07 15:56:36.592594 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:56:36.592568    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56b56968cc-gxml2" event={"ID":"fde2c89a-f756-424d-9630-8993c5eee62c","Type":"ContainerStarted","Data":"38a4cac322de1b47f74c782e62dfae47eb200f3308633ca9d43fec2f4bf7b50c"}
Jul 07 15:56:37.601358 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:56:37.601313    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56b56968cc-gxml2" event={"ID":"fde2c89a-f756-424d-9630-8993c5eee62c","Type":"ContainerStarted","Data":"3ec467e2f8a8ac0c51739fab9904b9acd4b61b87c9c39bcf1fbb90b39ccbcdd8"}
Jul 07 15:56:37.640827 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:56:37.640759    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-56b56968cc-gxml2" podStartSLOduration=2.300842173 podStartE2EDuration="2.640734689s" totalImagesPullingTime="339.892516ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:56:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:56:37.638659046 +0000 UTC m=+751.727084717" watchObservedRunningTime="2026-07-07 15:56:37.640734689 +0000 UTC m=+751.729160428"
Jul 07 15:56:37.694661 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:56:37.694628    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7f5d749cb4-qncrn"]
Jul 07 15:56:37.694934 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:56:37.694902    2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-7f5d749cb4-qncrn" podUID="ae1ed14a-cd55-40b6-a8e1-b567ef0dd040" containerName="authorino" containerID="cri-o://c64c9f498ef3c8187660aa6a88b9bc21a6158bf8f09fffd8e080007de085705c" gracePeriod=30
Jul 07 15:56:37.953110 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:56:37.953080    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7f5d749cb4-qncrn"
Jul 07 15:56:38.114590 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:56:38.114517    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ae1ed14a-cd55-40b6-a8e1-b567ef0dd040-tls-cert\") pod \"ae1ed14a-cd55-40b6-a8e1-b567ef0dd040\" (UID: \"ae1ed14a-cd55-40b6-a8e1-b567ef0dd040\") "
Jul 07 15:56:38.114741 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:56:38.114640    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vfjht\" (UniqueName: \"kubernetes.io/projected/ae1ed14a-cd55-40b6-a8e1-b567ef0dd040-kube-api-access-vfjht\") pod \"ae1ed14a-cd55-40b6-a8e1-b567ef0dd040\" (UID: \"ae1ed14a-cd55-40b6-a8e1-b567ef0dd040\") "
Jul 07 15:56:38.116636 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:56:38.116598    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae1ed14a-cd55-40b6-a8e1-b567ef0dd040-kube-api-access-vfjht" (OuterVolumeSpecName: "kube-api-access-vfjht") pod "ae1ed14a-cd55-40b6-a8e1-b567ef0dd040" (UID: "ae1ed14a-cd55-40b6-a8e1-b567ef0dd040"). InnerVolumeSpecName "kube-api-access-vfjht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 15:56:38.124926 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:56:38.124880    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae1ed14a-cd55-40b6-a8e1-b567ef0dd040-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "ae1ed14a-cd55-40b6-a8e1-b567ef0dd040" (UID: "ae1ed14a-cd55-40b6-a8e1-b567ef0dd040"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 15:56:38.215453 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:56:38.215414    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfjht\" (UniqueName: \"kubernetes.io/projected/ae1ed14a-cd55-40b6-a8e1-b567ef0dd040-kube-api-access-vfjht\") on node \"ip-10-0-138-101.ec2.internal\" DevicePath \"\""
Jul 07 15:56:38.215453 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:56:38.215450    2566 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ae1ed14a-cd55-40b6-a8e1-b567ef0dd040-tls-cert\") on node \"ip-10-0-138-101.ec2.internal\" DevicePath \"\""
Jul 07 15:56:38.607679 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:56:38.607641    2566 generic.go:356] "Generic (PLEG): container finished" podID="ae1ed14a-cd55-40b6-a8e1-b567ef0dd040" containerID="c64c9f498ef3c8187660aa6a88b9bc21a6158bf8f09fffd8e080007de085705c" exitCode=0
Jul 07 15:56:38.607679 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:56:38.607683    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7f5d749cb4-qncrn"
Jul 07 15:56:38.608195 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:56:38.607729    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7f5d749cb4-qncrn" event={"ID":"ae1ed14a-cd55-40b6-a8e1-b567ef0dd040","Type":"ContainerDied","Data":"c64c9f498ef3c8187660aa6a88b9bc21a6158bf8f09fffd8e080007de085705c"}
Jul 07 15:56:38.608195 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:56:38.607768    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7f5d749cb4-qncrn" event={"ID":"ae1ed14a-cd55-40b6-a8e1-b567ef0dd040","Type":"ContainerDied","Data":"411175616b55e11a6839e5610dc9f4cf7ce9563001ebcf863eb6745fb780fcab"}
Jul 07 15:56:38.608195 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:56:38.607788    2566 scope.go:117] "RemoveContainer" containerID="c64c9f498ef3c8187660aa6a88b9bc21a6158bf8f09fffd8e080007de085705c"
Jul 07 15:56:38.618598 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:56:38.618550    2566 scope.go:117] "RemoveContainer" containerID="c64c9f498ef3c8187660aa6a88b9bc21a6158bf8f09fffd8e080007de085705c"
Jul 07 15:56:38.619053 ip-10-0-138-101 kubenswrapper[2566]: E0707 15:56:38.619029    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c64c9f498ef3c8187660aa6a88b9bc21a6158bf8f09fffd8e080007de085705c\": container with ID starting with c64c9f498ef3c8187660aa6a88b9bc21a6158bf8f09fffd8e080007de085705c not found: ID does not exist" containerID="c64c9f498ef3c8187660aa6a88b9bc21a6158bf8f09fffd8e080007de085705c"
Jul 07 15:56:38.619146 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:56:38.619058    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c64c9f498ef3c8187660aa6a88b9bc21a6158bf8f09fffd8e080007de085705c"} err="failed to get container status \"c64c9f498ef3c8187660aa6a88b9bc21a6158bf8f09fffd8e080007de085705c\": rpc error: code = NotFound desc = could not find container \"c64c9f498ef3c8187660aa6a88b9bc21a6158bf8f09fffd8e080007de085705c\": container with ID starting with c64c9f498ef3c8187660aa6a88b9bc21a6158bf8f09fffd8e080007de085705c not found: ID does not exist"
Jul 07 15:56:38.623549 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:56:38.623528    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7f5d749cb4-qncrn"]
Jul 07 15:56:38.627900 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:56:38.627862    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7f5d749cb4-qncrn"]
Jul 07 15:56:40.474427 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:56:40.474397    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae1ed14a-cd55-40b6-a8e1-b567ef0dd040" path="/var/lib/kubelet/pods/ae1ed14a-cd55-40b6-a8e1-b567ef0dd040/volumes"
Jul 07 15:57:05.342923 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:57:05.342875    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mk99s"]
Jul 07 15:57:14.753940 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:57:14.753905    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mk99s"]
Jul 07 15:57:26.134771 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:57:26.134732    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mk99s"]
Jul 07 15:57:34.534623 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:57:34.534538    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mk99s"]
Jul 07 15:57:45.334690 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:57:45.334651    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mk99s"]
Jul 07 15:58:13.146785 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:58:13.146754    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mk99s"]
Jul 07 15:58:25.234206 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:58:25.234171    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mk99s"]
Jul 07 15:59:01.033254 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:59:01.033220    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mk99s"]
Jul 07 15:59:07.835100 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:59:07.835068    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mk99s"]
Jul 07 15:59:18.230992 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:59:18.230954    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mk99s"]
Jul 07 15:59:23.931452 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:59:23.931413    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mk99s"]
Jul 07 15:59:33.631464 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:59:33.631427    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mk99s"]
Jul 07 15:59:45.034526 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:59:45.034481    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mk99s"]
Jul 07 15:59:55.732921 ip-10-0-138-101 kubenswrapper[2566]: I0707 15:59:55.732866    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mk99s"]
Jul 07 16:00:05.135047 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:00:05.134999    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mk99s"]
Jul 07 16:00:28.133276 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:00:28.133238    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mk99s"]
Jul 07 16:00:37.429995 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:00:37.429955    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mk99s"]
Jul 07 16:00:59.433429 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:00:59.433392    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mk99s"]
Jul 07 16:01:09.329990 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:01:09.329951    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mk99s"]
Jul 07 16:01:25.240923 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:01:25.240873    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mk99s"]
Jul 07 16:01:35.031484 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:01:35.031444    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mk99s"]
Jul 07 16:01:45.428505 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:01:45.428467    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mk99s"]
Jul 07 16:01:55.038084 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:01:55.038051    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mk99s"]
Jul 07 16:02:04.032237 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:02:04.032196    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mk99s"]
Jul 07 16:02:30.123444 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:02:30.123405    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gvzb7"]
Jul 07 16:02:30.124045 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:02:30.123728    2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gvzb7" podUID="cc5263db-4499-40f7-8f0b-80f81d5e3a18" containerName="manager" containerID="cri-o://02444f6ec83e22dee4186c943d6cd16b865d112a58dcb4c6d41fd711f0561da9" gracePeriod=10
Jul 07 16:02:32.591852 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:02:32.591828    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gvzb7"
Jul 07 16:02:32.662915 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:02:32.662819    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tcts4\" (UniqueName: \"kubernetes.io/projected/cc5263db-4499-40f7-8f0b-80f81d5e3a18-kube-api-access-tcts4\") pod \"cc5263db-4499-40f7-8f0b-80f81d5e3a18\" (UID: \"cc5263db-4499-40f7-8f0b-80f81d5e3a18\") "
Jul 07 16:02:32.662915 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:02:32.662857    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cc5263db-4499-40f7-8f0b-80f81d5e3a18-extensions-socket-volume\") pod \"cc5263db-4499-40f7-8f0b-80f81d5e3a18\" (UID: \"cc5263db-4499-40f7-8f0b-80f81d5e3a18\") "
Jul 07 16:02:32.663363 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:02:32.663336    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc5263db-4499-40f7-8f0b-80f81d5e3a18-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "cc5263db-4499-40f7-8f0b-80f81d5e3a18" (UID: "cc5263db-4499-40f7-8f0b-80f81d5e3a18"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 16:02:32.664914 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:02:32.664867    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc5263db-4499-40f7-8f0b-80f81d5e3a18-kube-api-access-tcts4" (OuterVolumeSpecName: "kube-api-access-tcts4") pod "cc5263db-4499-40f7-8f0b-80f81d5e3a18" (UID: "cc5263db-4499-40f7-8f0b-80f81d5e3a18"). InnerVolumeSpecName "kube-api-access-tcts4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 16:02:32.764020 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:02:32.763982    2566 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cc5263db-4499-40f7-8f0b-80f81d5e3a18-extensions-socket-volume\") on node \"ip-10-0-138-101.ec2.internal\" DevicePath \"\""
Jul 07 16:02:32.764020 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:02:32.764017    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcts4\" (UniqueName: \"kubernetes.io/projected/cc5263db-4499-40f7-8f0b-80f81d5e3a18-kube-api-access-tcts4\") on node \"ip-10-0-138-101.ec2.internal\" DevicePath \"\""
Jul 07 16:02:33.249127 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:02:33.249094    2566 generic.go:356] "Generic (PLEG): container finished" podID="cc5263db-4499-40f7-8f0b-80f81d5e3a18" containerID="02444f6ec83e22dee4186c943d6cd16b865d112a58dcb4c6d41fd711f0561da9" exitCode=0
Jul 07 16:02:33.249294 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:02:33.249141    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gvzb7"
Jul 07 16:02:33.249294 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:02:33.249174    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gvzb7" event={"ID":"cc5263db-4499-40f7-8f0b-80f81d5e3a18","Type":"ContainerDied","Data":"02444f6ec83e22dee4186c943d6cd16b865d112a58dcb4c6d41fd711f0561da9"}
Jul 07 16:02:33.249294 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:02:33.249229    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gvzb7" event={"ID":"cc5263db-4499-40f7-8f0b-80f81d5e3a18","Type":"ContainerDied","Data":"63d0f200e720c331edad29033b8ef9f42db63789ffa4b8c762ae8e6e9c167e2f"}
Jul 07 16:02:33.249294 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:02:33.249259    2566 scope.go:117] "RemoveContainer" containerID="02444f6ec83e22dee4186c943d6cd16b865d112a58dcb4c6d41fd711f0561da9"
Jul 07 16:02:33.258599 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:02:33.258578    2566 scope.go:117] "RemoveContainer" containerID="02444f6ec83e22dee4186c943d6cd16b865d112a58dcb4c6d41fd711f0561da9"
Jul 07 16:02:33.258851 ip-10-0-138-101 kubenswrapper[2566]: E0707 16:02:33.258833    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02444f6ec83e22dee4186c943d6cd16b865d112a58dcb4c6d41fd711f0561da9\": container with ID starting with 02444f6ec83e22dee4186c943d6cd16b865d112a58dcb4c6d41fd711f0561da9 not found: ID does not exist" containerID="02444f6ec83e22dee4186c943d6cd16b865d112a58dcb4c6d41fd711f0561da9"
Jul 07 16:02:33.258942 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:02:33.258862    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02444f6ec83e22dee4186c943d6cd16b865d112a58dcb4c6d41fd711f0561da9"} err="failed to get container status \"02444f6ec83e22dee4186c943d6cd16b865d112a58dcb4c6d41fd711f0561da9\": rpc error: code = NotFound desc = could not find container \"02444f6ec83e22dee4186c943d6cd16b865d112a58dcb4c6d41fd711f0561da9\": container with ID starting with 02444f6ec83e22dee4186c943d6cd16b865d112a58dcb4c6d41fd711f0561da9 not found: ID does not exist"
Jul 07 16:02:33.273052 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:02:33.273028    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gvzb7"]
Jul 07 16:02:33.276927 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:02:33.276904    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gvzb7"]
Jul 07 16:02:34.474751 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:02:34.474721    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc5263db-4499-40f7-8f0b-80f81d5e3a18" path="/var/lib/kubelet/pods/cc5263db-4499-40f7-8f0b-80f81d5e3a18/volumes"
Jul 07 16:03:31.299645 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:03:31.299564    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n669q"]
Jul 07 16:03:31.300039 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:03:31.299951    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae1ed14a-cd55-40b6-a8e1-b567ef0dd040" containerName="authorino"
Jul 07 16:03:31.300039 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:03:31.299968    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae1ed14a-cd55-40b6-a8e1-b567ef0dd040" containerName="authorino"
Jul 07 16:03:31.300039 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:03:31.299996    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc5263db-4499-40f7-8f0b-80f81d5e3a18" containerName="manager"
Jul 07 16:03:31.300039 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:03:31.300002    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc5263db-4499-40f7-8f0b-80f81d5e3a18" containerName="manager"
Jul 07 16:03:31.300162 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:03:31.300051    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc5263db-4499-40f7-8f0b-80f81d5e3a18" containerName="manager"
Jul 07 16:03:31.300162 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:03:31.300061    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae1ed14a-cd55-40b6-a8e1-b567ef0dd040" containerName="authorino"
Jul 07 16:03:31.303964 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:03:31.303946    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n669q"
Jul 07 16:03:31.307717 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:03:31.307697    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-ttkqr\""
Jul 07 16:03:31.321563 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:03:31.321535    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n669q"]
Jul 07 16:03:31.366484 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:03:31.366448    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hksxm\" (UniqueName: \"kubernetes.io/projected/38eba44e-f8d4-4e90-9173-ee3251128848-kube-api-access-hksxm\") pod \"kuadrant-operator-controller-manager-86878c769f-n669q\" (UID: \"38eba44e-f8d4-4e90-9173-ee3251128848\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n669q"
Jul 07 16:03:31.366672 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:03:31.366491    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/38eba44e-f8d4-4e90-9173-ee3251128848-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-n669q\" (UID: \"38eba44e-f8d4-4e90-9173-ee3251128848\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n669q"
Jul 07 16:03:31.468008 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:03:31.467972    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hksxm\" (UniqueName: \"kubernetes.io/projected/38eba44e-f8d4-4e90-9173-ee3251128848-kube-api-access-hksxm\") pod \"kuadrant-operator-controller-manager-86878c769f-n669q\" (UID: \"38eba44e-f8d4-4e90-9173-ee3251128848\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n669q"
Jul 07 16:03:31.468198 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:03:31.468018    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/38eba44e-f8d4-4e90-9173-ee3251128848-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-n669q\" (UID: \"38eba44e-f8d4-4e90-9173-ee3251128848\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n669q"
Jul 07 16:03:31.468405 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:03:31.468373    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/38eba44e-f8d4-4e90-9173-ee3251128848-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-n669q\" (UID: \"38eba44e-f8d4-4e90-9173-ee3251128848\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n669q"
Jul 07 16:03:31.477715 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:03:31.477687    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hksxm\" (UniqueName: \"kubernetes.io/projected/38eba44e-f8d4-4e90-9173-ee3251128848-kube-api-access-hksxm\") pod \"kuadrant-operator-controller-manager-86878c769f-n669q\" (UID: \"38eba44e-f8d4-4e90-9173-ee3251128848\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n669q"
Jul 07 16:03:31.614473 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:03:31.614387    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n669q"
Jul 07 16:03:31.761287 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:03:31.761254    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n669q"]
Jul 07 16:03:31.764644 ip-10-0-138-101 kubenswrapper[2566]: W0707 16:03:31.764618    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38eba44e_f8d4_4e90_9173_ee3251128848.slice/crio-86863d0288ead7103bb44f19e3a75e06464c78cdd421fd08c0ce6c0a95484e68 WatchSource:0}: Error finding container 86863d0288ead7103bb44f19e3a75e06464c78cdd421fd08c0ce6c0a95484e68: Status 404 returned error can't find the container with id 86863d0288ead7103bb44f19e3a75e06464c78cdd421fd08c0ce6c0a95484e68
Jul 07 16:03:32.513369 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:03:32.513333    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n669q" event={"ID":"38eba44e-f8d4-4e90-9173-ee3251128848","Type":"ContainerStarted","Data":"796021d04b6bec0e81fef945deef13fd796b1fbb611733d5374cc2bb4381ff80"}
Jul 07 16:03:32.513798 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:03:32.513384    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n669q" event={"ID":"38eba44e-f8d4-4e90-9173-ee3251128848","Type":"ContainerStarted","Data":"86863d0288ead7103bb44f19e3a75e06464c78cdd421fd08c0ce6c0a95484e68"}
Jul 07 16:03:32.513798 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:03:32.513437    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n669q"
Jul 07 16:03:32.558261 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:03:32.558208    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n669q" podStartSLOduration=1.558190804 podStartE2EDuration="1.558190804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:03:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:03:32.557819245 +0000 UTC m=+1166.646244915" watchObservedRunningTime="2026-07-07 16:03:32.558190804 +0000 UTC m=+1166.646616473"
Jul 07 16:03:43.520027 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:03:43.519997    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n669q"
Jul 07 16:03:50.629486 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:03:50.629450    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mk99s"]
Jul 07 16:03:59.935949 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:03:59.935913    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mk99s"]
Jul 07 16:04:12.338260 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:04:12.338221    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mk99s"]
Jul 07 16:04:24.133227 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:04:24.133189    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mk99s"]
Jul 07 16:04:35.734481 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:04:35.734441    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mk99s"]
Jul 07 16:04:45.039709 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:04:45.039668    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mk99s"]
Jul 07 16:04:54.538774 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:04:54.538737    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mk99s"]
Jul 07 16:05:04.937758 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:05:04.937714    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mk99s"]
Jul 07 16:05:15.824490 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:05:15.824455    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mk99s"]
Jul 07 16:05:26.238430 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:05:26.238384    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mk99s"]
Jul 07 16:05:36.926232 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:05:36.926196    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mk99s"]
Jul 07 16:06:00.232523 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:06:00.232486    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mk99s"]
Jul 07 16:06:09.230746 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:06:09.230703    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mk99s"]
Jul 07 16:06:18.825716 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:06:18.825673    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mk99s"]
Jul 07 16:06:30.139203 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:06:30.139110    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mk99s"]
Jul 07 16:06:47.522659 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:06:47.522620    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mk99s"]
Jul 07 16:07:00.624217 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:07:00.624185    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mk99s"]
Jul 07 16:07:09.522927 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:07:09.522878    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mk99s"]
Jul 07 16:07:24.131973 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:07:24.131927    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mk99s"]
Jul 07 16:07:33.325282 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:07:33.325245    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mk99s"]
Jul 07 16:07:45.828697 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:07:45.828656    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mk99s"]
Jul 07 16:07:55.635732 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:07:55.635694    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mk99s"]
Jul 07 16:08:06.524712 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:08:06.524679    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mk99s"]
Jul 07 16:08:15.625191 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:08:15.625156    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mk99s"]
Jul 07 16:08:27.719877 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:08:27.719844    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mk99s"]
Jul 07 16:08:35.525279 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:08:35.525238    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mk99s"]
Jul 07 16:08:45.944782 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:08:45.944749    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mk99s"]
Jul 07 16:08:57.329139 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:08:57.329099    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mk99s"]
Jul 07 16:09:06.322444 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:09:06.322408    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mk99s"]
Jul 07 16:09:18.928661 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:09:18.928622    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mk99s"]
Jul 07 16:09:29.523519 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:09:29.523441    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mk99s"]
Jul 07 16:09:38.929205 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:09:38.929169    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mk99s"]
Jul 07 16:10:14.638561 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:10:14.638526    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mk99s"]
Jul 07 16:10:22.849092 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:10:22.849043    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mk99s"]
Jul 07 16:11:47.926578 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:11:47.926540    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mk99s"]
Jul 07 16:11:56.622743 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:11:56.622706    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mk99s"]
Jul 07 16:12:08.027296 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:12:08.027261    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mk99s"]
Jul 07 16:12:19.326080 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:12:19.326043    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mk99s"]
Jul 07 16:12:51.729340 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:12:51.729251    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mk99s"]
Jul 07 16:13:16.735276 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:13:16.735238    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-cadb2345/e2e-lifecycle-model-cadb2345-kserve-7d5fd66c7f-pgnfk"]
Jul 07 16:13:16.738610 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:13:16.738594    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-cadb2345/e2e-lifecycle-model-cadb2345-kserve-7d5fd66c7f-pgnfk"
Jul 07 16:13:16.741182 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:13:16.741159    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-cadb2345\"/\"kube-root-ca.crt\""
Jul 07 16:13:16.742274 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:13:16.742251    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-cadb2345\"/\"default-dockercfg-425pq\""
Jul 07 16:13:16.742388 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:13:16.742293    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-cadb2345\"/\"openshift-service-ca.crt\""
Jul 07 16:13:16.742388 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:13:16.742295    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-cadb2345\"/\"e2e-lifecycle-model-cadb2345-kserve-self-signed-certs\""
Jul 07 16:13:16.748837 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:13:16.748812    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-cadb2345/e2e-lifecycle-model-cadb2345-kserve-7d5fd66c7f-pgnfk"]
Jul 07 16:13:16.844776 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:13:16.844729    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/681f1c7e-3f3d-4c50-9792-e1f2b0391bbb-dshm\") pod \"e2e-lifecycle-model-cadb2345-kserve-7d5fd66c7f-pgnfk\" (UID: \"681f1c7e-3f3d-4c50-9792-e1f2b0391bbb\") " pod="ai-tenant-e2e-mt-cadb2345/e2e-lifecycle-model-cadb2345-kserve-7d5fd66c7f-pgnfk"
Jul 07 16:13:16.845005 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:13:16.844793    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/681f1c7e-3f3d-4c50-9792-e1f2b0391bbb-tmp-dir\") pod \"e2e-lifecycle-model-cadb2345-kserve-7d5fd66c7f-pgnfk\" (UID: \"681f1c7e-3f3d-4c50-9792-e1f2b0391bbb\") " pod="ai-tenant-e2e-mt-cadb2345/e2e-lifecycle-model-cadb2345-kserve-7d5fd66c7f-pgnfk"
Jul 07 16:13:16.845005 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:13:16.844822    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/681f1c7e-3f3d-4c50-9792-e1f2b0391bbb-tls-certs\") pod \"e2e-lifecycle-model-cadb2345-kserve-7d5fd66c7f-pgnfk\" (UID: \"681f1c7e-3f3d-4c50-9792-e1f2b0391bbb\") " pod="ai-tenant-e2e-mt-cadb2345/e2e-lifecycle-model-cadb2345-kserve-7d5fd66c7f-pgnfk"
Jul 07 16:13:16.845005 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:13:16.844849    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/681f1c7e-3f3d-4c50-9792-e1f2b0391bbb-home\") pod \"e2e-lifecycle-model-cadb2345-kserve-7d5fd66c7f-pgnfk\" (UID: \"681f1c7e-3f3d-4c50-9792-e1f2b0391bbb\") " pod="ai-tenant-e2e-mt-cadb2345/e2e-lifecycle-model-cadb2345-kserve-7d5fd66c7f-pgnfk"
Jul 07 16:13:16.845005 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:13:16.844871    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/681f1c7e-3f3d-4c50-9792-e1f2b0391bbb-model-cache\") pod \"e2e-lifecycle-model-cadb2345-kserve-7d5fd66c7f-pgnfk\" (UID: \"681f1c7e-3f3d-4c50-9792-e1f2b0391bbb\") " pod="ai-tenant-e2e-mt-cadb2345/e2e-lifecycle-model-cadb2345-kserve-7d5fd66c7f-pgnfk"
Jul 07 16:13:16.845005 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:13:16.844960    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-th66v\" (UniqueName: \"kubernetes.io/projected/681f1c7e-3f3d-4c50-9792-e1f2b0391bbb-kube-api-access-th66v\") pod \"e2e-lifecycle-model-cadb2345-kserve-7d5fd66c7f-pgnfk\" (UID: \"681f1c7e-3f3d-4c50-9792-e1f2b0391bbb\") " pod="ai-tenant-e2e-mt-cadb2345/e2e-lifecycle-model-cadb2345-kserve-7d5fd66c7f-pgnfk"
Jul 07 16:13:16.946174 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:13:16.946141    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-th66v\" (UniqueName: \"kubernetes.io/projected/681f1c7e-3f3d-4c50-9792-e1f2b0391bbb-kube-api-access-th66v\") pod \"e2e-lifecycle-model-cadb2345-kserve-7d5fd66c7f-pgnfk\" (UID: \"681f1c7e-3f3d-4c50-9792-e1f2b0391bbb\") " pod="ai-tenant-e2e-mt-cadb2345/e2e-lifecycle-model-cadb2345-kserve-7d5fd66c7f-pgnfk"
Jul 07 16:13:16.946368 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:13:16.946180    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/681f1c7e-3f3d-4c50-9792-e1f2b0391bbb-dshm\") pod \"e2e-lifecycle-model-cadb2345-kserve-7d5fd66c7f-pgnfk\" (UID: \"681f1c7e-3f3d-4c50-9792-e1f2b0391bbb\") " pod="ai-tenant-e2e-mt-cadb2345/e2e-lifecycle-model-cadb2345-kserve-7d5fd66c7f-pgnfk"
Jul 07 16:13:16.946368 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:13:16.946312    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/681f1c7e-3f3d-4c50-9792-e1f2b0391bbb-tmp-dir\") pod \"e2e-lifecycle-model-cadb2345-kserve-7d5fd66c7f-pgnfk\" (UID: \"681f1c7e-3f3d-4c50-9792-e1f2b0391bbb\") " pod="ai-tenant-e2e-mt-cadb2345/e2e-lifecycle-model-cadb2345-kserve-7d5fd66c7f-pgnfk"
Jul 07 16:13:16.946483 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:13:16.946369    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/681f1c7e-3f3d-4c50-9792-e1f2b0391bbb-tls-certs\") pod \"e2e-lifecycle-model-cadb2345-kserve-7d5fd66c7f-pgnfk\" (UID: \"681f1c7e-3f3d-4c50-9792-e1f2b0391bbb\") " pod="ai-tenant-e2e-mt-cadb2345/e2e-lifecycle-model-cadb2345-kserve-7d5fd66c7f-pgnfk"
Jul 07 16:13:16.946483 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:13:16.946411    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/681f1c7e-3f3d-4c50-9792-e1f2b0391bbb-home\") pod \"e2e-lifecycle-model-cadb2345-kserve-7d5fd66c7f-pgnfk\" (UID: \"681f1c7e-3f3d-4c50-9792-e1f2b0391bbb\") " pod="ai-tenant-e2e-mt-cadb2345/e2e-lifecycle-model-cadb2345-kserve-7d5fd66c7f-pgnfk"
Jul 07 16:13:16.946483 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:13:16.946436    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/681f1c7e-3f3d-4c50-9792-e1f2b0391bbb-model-cache\") pod \"e2e-lifecycle-model-cadb2345-kserve-7d5fd66c7f-pgnfk\" (UID: \"681f1c7e-3f3d-4c50-9792-e1f2b0391bbb\") " pod="ai-tenant-e2e-mt-cadb2345/e2e-lifecycle-model-cadb2345-kserve-7d5fd66c7f-pgnfk"
Jul 07 16:13:16.946735 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:13:16.946710    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/681f1c7e-3f3d-4c50-9792-e1f2b0391bbb-tmp-dir\") pod \"e2e-lifecycle-model-cadb2345-kserve-7d5fd66c7f-pgnfk\" (UID: \"681f1c7e-3f3d-4c50-9792-e1f2b0391bbb\") " pod="ai-tenant-e2e-mt-cadb2345/e2e-lifecycle-model-cadb2345-kserve-7d5fd66c7f-pgnfk"
Jul 07 16:13:16.946850 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:13:16.946780    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/681f1c7e-3f3d-4c50-9792-e1f2b0391bbb-home\") pod \"e2e-lifecycle-model-cadb2345-kserve-7d5fd66c7f-pgnfk\" (UID: \"681f1c7e-3f3d-4c50-9792-e1f2b0391bbb\") " pod="ai-tenant-e2e-mt-cadb2345/e2e-lifecycle-model-cadb2345-kserve-7d5fd66c7f-pgnfk"
Jul 07 16:13:16.946850 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:13:16.946824    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/681f1c7e-3f3d-4c50-9792-e1f2b0391bbb-model-cache\") pod \"e2e-lifecycle-model-cadb2345-kserve-7d5fd66c7f-pgnfk\" (UID: \"681f1c7e-3f3d-4c50-9792-e1f2b0391bbb\") " pod="ai-tenant-e2e-mt-cadb2345/e2e-lifecycle-model-cadb2345-kserve-7d5fd66c7f-pgnfk"
Jul 07 16:13:16.948441 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:13:16.948389    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/681f1c7e-3f3d-4c50-9792-e1f2b0391bbb-dshm\") pod \"e2e-lifecycle-model-cadb2345-kserve-7d5fd66c7f-pgnfk\" (UID: \"681f1c7e-3f3d-4c50-9792-e1f2b0391bbb\") " pod="ai-tenant-e2e-mt-cadb2345/e2e-lifecycle-model-cadb2345-kserve-7d5fd66c7f-pgnfk"
Jul 07 16:13:16.948899 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:13:16.948865    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/681f1c7e-3f3d-4c50-9792-e1f2b0391bbb-tls-certs\") pod \"e2e-lifecycle-model-cadb2345-kserve-7d5fd66c7f-pgnfk\" (UID: \"681f1c7e-3f3d-4c50-9792-e1f2b0391bbb\") " pod="ai-tenant-e2e-mt-cadb2345/e2e-lifecycle-model-cadb2345-kserve-7d5fd66c7f-pgnfk"
Jul 07 16:13:16.955467 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:13:16.955443    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-th66v\" (UniqueName: \"kubernetes.io/projected/681f1c7e-3f3d-4c50-9792-e1f2b0391bbb-kube-api-access-th66v\") pod \"e2e-lifecycle-model-cadb2345-kserve-7d5fd66c7f-pgnfk\" (UID: \"681f1c7e-3f3d-4c50-9792-e1f2b0391bbb\") " pod="ai-tenant-e2e-mt-cadb2345/e2e-lifecycle-model-cadb2345-kserve-7d5fd66c7f-pgnfk"
Jul 07 16:13:17.047775 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:13:17.047687    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-cadb2345/e2e-lifecycle-model-cadb2345-kserve-7d5fd66c7f-pgnfk"
Jul 07 16:13:17.189810 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:13:17.189774    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-cadb2345/e2e-lifecycle-model-cadb2345-kserve-7d5fd66c7f-pgnfk"]
Jul 07 16:13:17.192918 ip-10-0-138-101 kubenswrapper[2566]: W0707 16:13:17.192865    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod681f1c7e_3f3d_4c50_9792_e1f2b0391bbb.slice/crio-4ac6d1065b0c4c9458c1c43c6270ea52835c67a010c4bd4b0fcb483a573b471e WatchSource:0}: Error finding container 4ac6d1065b0c4c9458c1c43c6270ea52835c67a010c4bd4b0fcb483a573b471e: Status 404 returned error can't find the container with id 4ac6d1065b0c4c9458c1c43c6270ea52835c67a010c4bd4b0fcb483a573b471e
Jul 07 16:13:18.154835 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:13:18.154796    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-cadb2345/e2e-lifecycle-model-cadb2345-kserve-7d5fd66c7f-pgnfk" event={"ID":"681f1c7e-3f3d-4c50-9792-e1f2b0391bbb","Type":"ContainerStarted","Data":"b8d43d3bb86dd551bd65f0ccc2f1ee3390b8bde150fea3603cacf8aa1f3be4b5"}
Jul 07 16:13:18.154835 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:13:18.154837    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-cadb2345/e2e-lifecycle-model-cadb2345-kserve-7d5fd66c7f-pgnfk" event={"ID":"681f1c7e-3f3d-4c50-9792-e1f2b0391bbb","Type":"ContainerStarted","Data":"4ac6d1065b0c4c9458c1c43c6270ea52835c67a010c4bd4b0fcb483a573b471e"}
Jul 07 16:13:18.174857 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:13:18.174805    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-cadb2345/e2e-lifecycle-model-cadb2345-kserve-7d5fd66c7f-pgnfk" podStartSLOduration=2.174790553 podStartE2EDuration="2.174790553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:13:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:13:18.172441303 +0000 UTC m=+1752.260866972" watchObservedRunningTime="2026-07-07 16:13:18.174790553 +0000 UTC m=+1752.263216223"
Jul 07 16:13:20.230427 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:13:20.230390    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mk99s"]
Jul 07 16:13:27.047828 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:13:27.047784    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-cadb2345/e2e-lifecycle-model-cadb2345-kserve-7d5fd66c7f-pgnfk"
Jul 07 16:13:27.047828 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:13:27.047841    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-cadb2345/e2e-lifecycle-model-cadb2345-kserve-7d5fd66c7f-pgnfk"
Jul 07 16:13:27.060512 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:13:27.060484    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-cadb2345/e2e-lifecycle-model-cadb2345-kserve-7d5fd66c7f-pgnfk"
Jul 07 16:13:27.209289 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:13:27.209262    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-cadb2345/e2e-lifecycle-model-cadb2345-kserve-7d5fd66c7f-pgnfk"
Jul 07 16:13:36.624303 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:13:36.624268    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mk99s"]
Jul 07 16:13:50.067319 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:13:50.067278    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-cadb2345/e2e-lifecycle-model-cadb2345-kserve-7d5fd66c7f-pgnfk"]
Jul 07 16:13:50.224977 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:13:50.224944    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mk99s"]
Jul 07 16:14:05.088146 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:14:05.088102    2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-mt-cadb2345/e2e-lifecycle-model-cadb2345-kserve-7d5fd66c7f-pgnfk" podUID="681f1c7e-3f3d-4c50-9792-e1f2b0391bbb" containerName="main" containerID="cri-o://b8d43d3bb86dd551bd65f0ccc2f1ee3390b8bde150fea3603cacf8aa1f3be4b5" gracePeriod=45
Jul 07 16:14:05.341727 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:14:05.341665    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-cadb2345/e2e-lifecycle-model-cadb2345-kserve-7d5fd66c7f-pgnfk"
Jul 07 16:14:05.370650 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:14:05.370616    2566 generic.go:356] "Generic (PLEG): container finished" podID="681f1c7e-3f3d-4c50-9792-e1f2b0391bbb" containerID="b8d43d3bb86dd551bd65f0ccc2f1ee3390b8bde150fea3603cacf8aa1f3be4b5" exitCode=0
Jul 07 16:14:05.370811 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:14:05.370657    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-cadb2345/e2e-lifecycle-model-cadb2345-kserve-7d5fd66c7f-pgnfk"
Jul 07 16:14:05.370811 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:14:05.370674    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-cadb2345/e2e-lifecycle-model-cadb2345-kserve-7d5fd66c7f-pgnfk" event={"ID":"681f1c7e-3f3d-4c50-9792-e1f2b0391bbb","Type":"ContainerDied","Data":"b8d43d3bb86dd551bd65f0ccc2f1ee3390b8bde150fea3603cacf8aa1f3be4b5"}
Jul 07 16:14:05.370811 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:14:05.370713    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-cadb2345/e2e-lifecycle-model-cadb2345-kserve-7d5fd66c7f-pgnfk" event={"ID":"681f1c7e-3f3d-4c50-9792-e1f2b0391bbb","Type":"ContainerDied","Data":"4ac6d1065b0c4c9458c1c43c6270ea52835c67a010c4bd4b0fcb483a573b471e"}
Jul 07 16:14:05.370811 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:14:05.370735    2566 scope.go:117] "RemoveContainer" containerID="b8d43d3bb86dd551bd65f0ccc2f1ee3390b8bde150fea3603cacf8aa1f3be4b5"
Jul 07 16:14:05.378789 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:14:05.378769    2566 scope.go:117] "RemoveContainer" containerID="b8d43d3bb86dd551bd65f0ccc2f1ee3390b8bde150fea3603cacf8aa1f3be4b5"
Jul 07 16:14:05.379075 ip-10-0-138-101 kubenswrapper[2566]: E0707 16:14:05.379055    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8d43d3bb86dd551bd65f0ccc2f1ee3390b8bde150fea3603cacf8aa1f3be4b5\": container with ID starting with b8d43d3bb86dd551bd65f0ccc2f1ee3390b8bde150fea3603cacf8aa1f3be4b5 not found: ID does not exist" containerID="b8d43d3bb86dd551bd65f0ccc2f1ee3390b8bde150fea3603cacf8aa1f3be4b5"
Jul 07 16:14:05.379138 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:14:05.379084    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8d43d3bb86dd551bd65f0ccc2f1ee3390b8bde150fea3603cacf8aa1f3be4b5"} err="failed to get container status \"b8d43d3bb86dd551bd65f0ccc2f1ee3390b8bde150fea3603cacf8aa1f3be4b5\": rpc error: code = NotFound desc = could not find container \"b8d43d3bb86dd551bd65f0ccc2f1ee3390b8bde150fea3603cacf8aa1f3be4b5\": container with ID starting with b8d43d3bb86dd551bd65f0ccc2f1ee3390b8bde150fea3603cacf8aa1f3be4b5 not found: ID does not exist"
Jul 07 16:14:05.438483 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:14:05.438454    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/681f1c7e-3f3d-4c50-9792-e1f2b0391bbb-tmp-dir\") pod \"681f1c7e-3f3d-4c50-9792-e1f2b0391bbb\" (UID: \"681f1c7e-3f3d-4c50-9792-e1f2b0391bbb\") "
Jul 07 16:14:05.438632 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:14:05.438493    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/681f1c7e-3f3d-4c50-9792-e1f2b0391bbb-dshm\") pod \"681f1c7e-3f3d-4c50-9792-e1f2b0391bbb\" (UID: \"681f1c7e-3f3d-4c50-9792-e1f2b0391bbb\") "
Jul 07 16:14:05.438632 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:14:05.438518    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/681f1c7e-3f3d-4c50-9792-e1f2b0391bbb-tls-certs\") pod \"681f1c7e-3f3d-4c50-9792-e1f2b0391bbb\" (UID: \"681f1c7e-3f3d-4c50-9792-e1f2b0391bbb\") "
Jul 07 16:14:05.438632 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:14:05.438564    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/681f1c7e-3f3d-4c50-9792-e1f2b0391bbb-model-cache\") pod \"681f1c7e-3f3d-4c50-9792-e1f2b0391bbb\" (UID: \"681f1c7e-3f3d-4c50-9792-e1f2b0391bbb\") "
Jul 07 16:14:05.438632 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:14:05.438590    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-th66v\" (UniqueName: \"kubernetes.io/projected/681f1c7e-3f3d-4c50-9792-e1f2b0391bbb-kube-api-access-th66v\") pod \"681f1c7e-3f3d-4c50-9792-e1f2b0391bbb\" (UID: \"681f1c7e-3f3d-4c50-9792-e1f2b0391bbb\") "
Jul 07 16:14:05.438632 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:14:05.438617    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/681f1c7e-3f3d-4c50-9792-e1f2b0391bbb-home\") pod \"681f1c7e-3f3d-4c50-9792-e1f2b0391bbb\" (UID: \"681f1c7e-3f3d-4c50-9792-e1f2b0391bbb\") "
Jul 07 16:14:05.438928 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:14:05.438783    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/681f1c7e-3f3d-4c50-9792-e1f2b0391bbb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "681f1c7e-3f3d-4c50-9792-e1f2b0391bbb" (UID: "681f1c7e-3f3d-4c50-9792-e1f2b0391bbb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 16:14:05.438928 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:14:05.438911    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/681f1c7e-3f3d-4c50-9792-e1f2b0391bbb-tmp-dir\") on node \"ip-10-0-138-101.ec2.internal\" DevicePath \"\""
Jul 07 16:14:05.439044 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:14:05.438917    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/681f1c7e-3f3d-4c50-9792-e1f2b0391bbb-model-cache" (OuterVolumeSpecName: "model-cache") pod "681f1c7e-3f3d-4c50-9792-e1f2b0391bbb" (UID: "681f1c7e-3f3d-4c50-9792-e1f2b0391bbb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 16:14:05.439044 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:14:05.439021    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/681f1c7e-3f3d-4c50-9792-e1f2b0391bbb-home" (OuterVolumeSpecName: "home") pod "681f1c7e-3f3d-4c50-9792-e1f2b0391bbb" (UID: "681f1c7e-3f3d-4c50-9792-e1f2b0391bbb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 16:14:05.440834 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:14:05.440802    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/681f1c7e-3f3d-4c50-9792-e1f2b0391bbb-kube-api-access-th66v" (OuterVolumeSpecName: "kube-api-access-th66v") pod "681f1c7e-3f3d-4c50-9792-e1f2b0391bbb" (UID: "681f1c7e-3f3d-4c50-9792-e1f2b0391bbb"). InnerVolumeSpecName "kube-api-access-th66v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 16:14:05.440834 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:14:05.440828    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/681f1c7e-3f3d-4c50-9792-e1f2b0391bbb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "681f1c7e-3f3d-4c50-9792-e1f2b0391bbb" (UID: "681f1c7e-3f3d-4c50-9792-e1f2b0391bbb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 16:14:05.440991 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:14:05.440872    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/681f1c7e-3f3d-4c50-9792-e1f2b0391bbb-dshm" (OuterVolumeSpecName: "dshm") pod "681f1c7e-3f3d-4c50-9792-e1f2b0391bbb" (UID: "681f1c7e-3f3d-4c50-9792-e1f2b0391bbb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 16:14:05.540285 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:14:05.540250    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/681f1c7e-3f3d-4c50-9792-e1f2b0391bbb-model-cache\") on node \"ip-10-0-138-101.ec2.internal\" DevicePath \"\""
Jul 07 16:14:05.540285 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:14:05.540281    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-th66v\" (UniqueName: \"kubernetes.io/projected/681f1c7e-3f3d-4c50-9792-e1f2b0391bbb-kube-api-access-th66v\") on node \"ip-10-0-138-101.ec2.internal\" DevicePath \"\""
Jul 07 16:14:05.540285 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:14:05.540291    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/681f1c7e-3f3d-4c50-9792-e1f2b0391bbb-home\") on node \"ip-10-0-138-101.ec2.internal\" DevicePath \"\""
Jul 07 16:14:05.540496 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:14:05.540301    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/681f1c7e-3f3d-4c50-9792-e1f2b0391bbb-dshm\") on node \"ip-10-0-138-101.ec2.internal\" DevicePath \"\""
Jul 07 16:14:05.540496 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:14:05.540311    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/681f1c7e-3f3d-4c50-9792-e1f2b0391bbb-tls-certs\") on node \"ip-10-0-138-101.ec2.internal\" DevicePath \"\""
Jul 07 16:14:05.691264 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:14:05.691233    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-cadb2345/e2e-lifecycle-model-cadb2345-kserve-7d5fd66c7f-pgnfk"]
Jul 07 16:14:05.695143 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:14:05.695120    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-cadb2345/e2e-lifecycle-model-cadb2345-kserve-7d5fd66c7f-pgnfk"]
Jul 07 16:14:06.475997 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:14:06.475961    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="681f1c7e-3f3d-4c50-9792-e1f2b0391bbb" path="/var/lib/kubelet/pods/681f1c7e-3f3d-4c50-9792-e1f2b0391bbb/volumes"
Jul 07 16:14:30.236437 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:14:30.236400    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mk99s"]
Jul 07 16:14:38.933657 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:14:38.933622    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mk99s"]
Jul 07 16:14:51.845364 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:14:51.845326    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mk99s"]
Jul 07 16:15:38.740704 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:15:38.740632    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-76e259-openshift-default-fcb796d6c-9k9p9"]
Jul 07 16:15:38.741128 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:15:38.740987    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="681f1c7e-3f3d-4c50-9792-e1f2b0391bbb" containerName="main"
Jul 07 16:15:38.741128 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:15:38.741000    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="681f1c7e-3f3d-4c50-9792-e1f2b0391bbb" containerName="main"
Jul 07 16:15:38.741128 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:15:38.741071    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="681f1c7e-3f3d-4c50-9792-e1f2b0391bbb" containerName="main"
Jul 07 16:15:38.743976 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:15:38.743955    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-76e259-openshift-default-fcb796d6c-9k9p9"
Jul 07 16:15:38.746185 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:15:38.746165    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-76e259-openshift-default-dockercfg-xtmlv\""
Jul 07 16:15:38.762188 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:15:38.762163    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-76e259-openshift-default-fcb796d6c-9k9p9"]
Jul 07 16:15:38.838219 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:15:38.838185    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7r5kb\" (UniqueName: \"kubernetes.io/projected/cebc66f0-4d97-41bd-acb3-73ddbbd88260-kube-api-access-7r5kb\") pod \"e2e-inf-b-76e259-openshift-default-fcb796d6c-9k9p9\" (UID: \"cebc66f0-4d97-41bd-acb3-73ddbbd88260\") " pod="openshift-ingress/e2e-inf-b-76e259-openshift-default-fcb796d6c-9k9p9"
Jul 07 16:15:38.838406 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:15:38.838227    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cebc66f0-4d97-41bd-acb3-73ddbbd88260-istio-data\") pod \"e2e-inf-b-76e259-openshift-default-fcb796d6c-9k9p9\" (UID: \"cebc66f0-4d97-41bd-acb3-73ddbbd88260\") " pod="openshift-ingress/e2e-inf-b-76e259-openshift-default-fcb796d6c-9k9p9"
Jul 07 16:15:38.838406 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:15:38.838292    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cebc66f0-4d97-41bd-acb3-73ddbbd88260-istiod-ca-cert\") pod \"e2e-inf-b-76e259-openshift-default-fcb796d6c-9k9p9\" (UID: \"cebc66f0-4d97-41bd-acb3-73ddbbd88260\") " pod="openshift-ingress/e2e-inf-b-76e259-openshift-default-fcb796d6c-9k9p9"
Jul 07 16:15:38.838406 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:15:38.838320    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cebc66f0-4d97-41bd-acb3-73ddbbd88260-workload-socket\") pod \"e2e-inf-b-76e259-openshift-default-fcb796d6c-9k9p9\" (UID: \"cebc66f0-4d97-41bd-acb3-73ddbbd88260\") " pod="openshift-ingress/e2e-inf-b-76e259-openshift-default-fcb796d6c-9k9p9"
Jul 07 16:15:38.838406 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:15:38.838397    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cebc66f0-4d97-41bd-acb3-73ddbbd88260-workload-certs\") pod \"e2e-inf-b-76e259-openshift-default-fcb796d6c-9k9p9\" (UID: \"cebc66f0-4d97-41bd-acb3-73ddbbd88260\") " pod="openshift-ingress/e2e-inf-b-76e259-openshift-default-fcb796d6c-9k9p9"
Jul 07 16:15:38.838621 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:15:38.838437    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cebc66f0-4d97-41bd-acb3-73ddbbd88260-istio-envoy\") pod \"e2e-inf-b-76e259-openshift-default-fcb796d6c-9k9p9\" (UID: \"cebc66f0-4d97-41bd-acb3-73ddbbd88260\") " pod="openshift-ingress/e2e-inf-b-76e259-openshift-default-fcb796d6c-9k9p9"
Jul 07 16:15:38.838621 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:15:38.838493    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cebc66f0-4d97-41bd-acb3-73ddbbd88260-istio-podinfo\") pod \"e2e-inf-b-76e259-openshift-default-fcb796d6c-9k9p9\" (UID: \"cebc66f0-4d97-41bd-acb3-73ddbbd88260\") " pod="openshift-ingress/e2e-inf-b-76e259-openshift-default-fcb796d6c-9k9p9"
Jul 07 16:15:38.838621 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:15:38.838516    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cebc66f0-4d97-41bd-acb3-73ddbbd88260-istio-token\") pod \"e2e-inf-b-76e259-openshift-default-fcb796d6c-9k9p9\" (UID: \"cebc66f0-4d97-41bd-acb3-73ddbbd88260\") " pod="openshift-ingress/e2e-inf-b-76e259-openshift-default-fcb796d6c-9k9p9"
Jul 07 16:15:38.838621 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:15:38.838561    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cebc66f0-4d97-41bd-acb3-73ddbbd88260-credential-socket\") pod \"e2e-inf-b-76e259-openshift-default-fcb796d6c-9k9p9\" (UID: \"cebc66f0-4d97-41bd-acb3-73ddbbd88260\") " pod="openshift-ingress/e2e-inf-b-76e259-openshift-default-fcb796d6c-9k9p9"
Jul 07 16:15:38.939273 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:15:38.939242    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cebc66f0-4d97-41bd-acb3-73ddbbd88260-istiod-ca-cert\") pod \"e2e-inf-b-76e259-openshift-default-fcb796d6c-9k9p9\" (UID: \"cebc66f0-4d97-41bd-acb3-73ddbbd88260\") " pod="openshift-ingress/e2e-inf-b-76e259-openshift-default-fcb796d6c-9k9p9"
Jul 07 16:15:38.939455 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:15:38.939281    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cebc66f0-4d97-41bd-acb3-73ddbbd88260-workload-socket\") pod \"e2e-inf-b-76e259-openshift-default-fcb796d6c-9k9p9\" (UID: \"cebc66f0-4d97-41bd-acb3-73ddbbd88260\") " pod="openshift-ingress/e2e-inf-b-76e259-openshift-default-fcb796d6c-9k9p9"
Jul 07 16:15:38.939455 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:15:38.939325    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cebc66f0-4d97-41bd-acb3-73ddbbd88260-workload-certs\") pod \"e2e-inf-b-76e259-openshift-default-fcb796d6c-9k9p9\" (UID: \"cebc66f0-4d97-41bd-acb3-73ddbbd88260\") " pod="openshift-ingress/e2e-inf-b-76e259-openshift-default-fcb796d6c-9k9p9"
Jul 07 16:15:38.939569 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:15:38.939452    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cebc66f0-4d97-41bd-acb3-73ddbbd88260-istio-envoy\") pod \"e2e-inf-b-76e259-openshift-default-fcb796d6c-9k9p9\" (UID: \"cebc66f0-4d97-41bd-acb3-73ddbbd88260\") " pod="openshift-ingress/e2e-inf-b-76e259-openshift-default-fcb796d6c-9k9p9"
Jul 07 16:15:38.939569 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:15:38.939516    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cebc66f0-4d97-41bd-acb3-73ddbbd88260-istio-podinfo\") pod \"e2e-inf-b-76e259-openshift-default-fcb796d6c-9k9p9\" (UID: \"cebc66f0-4d97-41bd-acb3-73ddbbd88260\") " pod="openshift-ingress/e2e-inf-b-76e259-openshift-default-fcb796d6c-9k9p9"
Jul 07 16:15:38.939569 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:15:38.939544    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cebc66f0-4d97-41bd-acb3-73ddbbd88260-istio-token\") pod \"e2e-inf-b-76e259-openshift-default-fcb796d6c-9k9p9\" (UID: \"cebc66f0-4d97-41bd-acb3-73ddbbd88260\") " pod="openshift-ingress/e2e-inf-b-76e259-openshift-default-fcb796d6c-9k9p9"
Jul 07 16:15:38.939715 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:15:38.939604    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cebc66f0-4d97-41bd-acb3-73ddbbd88260-credential-socket\") pod \"e2e-inf-b-76e259-openshift-default-fcb796d6c-9k9p9\" (UID: \"cebc66f0-4d97-41bd-acb3-73ddbbd88260\") " pod="openshift-ingress/e2e-inf-b-76e259-openshift-default-fcb796d6c-9k9p9"
Jul 07 16:15:38.939715 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:15:38.939648    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7r5kb\" (UniqueName: \"kubernetes.io/projected/cebc66f0-4d97-41bd-acb3-73ddbbd88260-kube-api-access-7r5kb\") pod \"e2e-inf-b-76e259-openshift-default-fcb796d6c-9k9p9\" (UID: \"cebc66f0-4d97-41bd-acb3-73ddbbd88260\") " pod="openshift-ingress/e2e-inf-b-76e259-openshift-default-fcb796d6c-9k9p9"
Jul 07 16:15:38.939715 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:15:38.939688    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cebc66f0-4d97-41bd-acb3-73ddbbd88260-istio-data\") pod \"e2e-inf-b-76e259-openshift-default-fcb796d6c-9k9p9\" (UID: \"cebc66f0-4d97-41bd-acb3-73ddbbd88260\") " pod="openshift-ingress/e2e-inf-b-76e259-openshift-default-fcb796d6c-9k9p9"
Jul 07 16:15:38.939901 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:15:38.939862    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cebc66f0-4d97-41bd-acb3-73ddbbd88260-workload-socket\") pod \"e2e-inf-b-76e259-openshift-default-fcb796d6c-9k9p9\" (UID: \"cebc66f0-4d97-41bd-acb3-73ddbbd88260\") " pod="openshift-ingress/e2e-inf-b-76e259-openshift-default-fcb796d6c-9k9p9"
Jul 07 16:15:38.939958 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:15:38.939931    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cebc66f0-4d97-41bd-acb3-73ddbbd88260-workload-certs\") pod \"e2e-inf-b-76e259-openshift-default-fcb796d6c-9k9p9\" (UID: \"cebc66f0-4d97-41bd-acb3-73ddbbd88260\") " pod="openshift-ingress/e2e-inf-b-76e259-openshift-default-fcb796d6c-9k9p9"
Jul 07 16:15:38.940245 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:15:38.940218    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cebc66f0-4d97-41bd-acb3-73ddbbd88260-credential-socket\") pod \"e2e-inf-b-76e259-openshift-default-fcb796d6c-9k9p9\" (UID: \"cebc66f0-4d97-41bd-acb3-73ddbbd88260\") " pod="openshift-ingress/e2e-inf-b-76e259-openshift-default-fcb796d6c-9k9p9"
Jul 07 16:15:38.940385 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:15:38.940254    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cebc66f0-4d97-41bd-acb3-73ddbbd88260-istiod-ca-cert\") pod \"e2e-inf-b-76e259-openshift-default-fcb796d6c-9k9p9\" (UID: \"cebc66f0-4d97-41bd-acb3-73ddbbd88260\") " pod="openshift-ingress/e2e-inf-b-76e259-openshift-default-fcb796d6c-9k9p9"
Jul 07 16:15:38.940385 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:15:38.940287    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cebc66f0-4d97-41bd-acb3-73ddbbd88260-istio-data\") pod \"e2e-inf-b-76e259-openshift-default-fcb796d6c-9k9p9\" (UID: \"cebc66f0-4d97-41bd-acb3-73ddbbd88260\") " pod="openshift-ingress/e2e-inf-b-76e259-openshift-default-fcb796d6c-9k9p9"
Jul 07 16:15:38.942031 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:15:38.942008    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cebc66f0-4d97-41bd-acb3-73ddbbd88260-istio-podinfo\") pod \"e2e-inf-b-76e259-openshift-default-fcb796d6c-9k9p9\" (UID: \"cebc66f0-4d97-41bd-acb3-73ddbbd88260\") " pod="openshift-ingress/e2e-inf-b-76e259-openshift-default-fcb796d6c-9k9p9"
Jul 07 16:15:38.942128 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:15:38.942013    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cebc66f0-4d97-41bd-acb3-73ddbbd88260-istio-envoy\") pod \"e2e-inf-b-76e259-openshift-default-fcb796d6c-9k9p9\" (UID: \"cebc66f0-4d97-41bd-acb3-73ddbbd88260\") " pod="openshift-ingress/e2e-inf-b-76e259-openshift-default-fcb796d6c-9k9p9"
Jul 07 16:15:38.951126 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:15:38.951100    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cebc66f0-4d97-41bd-acb3-73ddbbd88260-istio-token\") pod \"e2e-inf-b-76e259-openshift-default-fcb796d6c-9k9p9\" (UID: \"cebc66f0-4d97-41bd-acb3-73ddbbd88260\") " pod="openshift-ingress/e2e-inf-b-76e259-openshift-default-fcb796d6c-9k9p9"
Jul 07 16:15:38.951263 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:15:38.951225    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7r5kb\" (UniqueName: \"kubernetes.io/projected/cebc66f0-4d97-41bd-acb3-73ddbbd88260-kube-api-access-7r5kb\") pod \"e2e-inf-b-76e259-openshift-default-fcb796d6c-9k9p9\" (UID: \"cebc66f0-4d97-41bd-acb3-73ddbbd88260\") " pod="openshift-ingress/e2e-inf-b-76e259-openshift-default-fcb796d6c-9k9p9"
Jul 07 16:15:39.054358 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:15:39.054271    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-76e259-openshift-default-fcb796d6c-9k9p9"
Jul 07 16:15:39.195576 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:15:39.195548    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-76e259-openshift-default-fcb796d6c-9k9p9"]
Jul 07 16:15:39.197704 ip-10-0-138-101 kubenswrapper[2566]: W0707 16:15:39.197671    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcebc66f0_4d97_41bd_acb3_73ddbbd88260.slice/crio-b07c6e3e1a7f18bb7d444d3e26c10b187669e1b3cbdc5bdc5514ecc474a42ade WatchSource:0}: Error finding container b07c6e3e1a7f18bb7d444d3e26c10b187669e1b3cbdc5bdc5514ecc474a42ade: Status 404 returned error can't find the container with id b07c6e3e1a7f18bb7d444d3e26c10b187669e1b3cbdc5bdc5514ecc474a42ade
Jul 07 16:15:39.199714 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:15:39.199684    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 07 16:15:39.199780 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:15:39.199752    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 07 16:15:39.199816 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:15:39.199778    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 07 16:15:39.796237 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:15:39.796205    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-76e259-openshift-default-fcb796d6c-9k9p9" event={"ID":"cebc66f0-4d97-41bd-acb3-73ddbbd88260","Type":"ContainerStarted","Data":"b33f8945a6a4c912602847a245a333e3ece59768806d0bdfee08cc2597c6b309"}
Jul 07 16:15:39.796617 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:15:39.796284    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-76e259-openshift-default-fcb796d6c-9k9p9" event={"ID":"cebc66f0-4d97-41bd-acb3-73ddbbd88260","Type":"ContainerStarted","Data":"b07c6e3e1a7f18bb7d444d3e26c10b187669e1b3cbdc5bdc5514ecc474a42ade"}
Jul 07 16:15:39.820504 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:15:39.820442    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-76e259-openshift-default-fcb796d6c-9k9p9" podStartSLOduration=1.820419429 podStartE2EDuration="1.820419429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:15:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:15:39.819101423 +0000 UTC m=+1893.907527096" watchObservedRunningTime="2026-07-07 16:15:39.820419429 +0000 UTC m=+1893.908845100"
Jul 07 16:15:40.055133 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:15:40.055034    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-76e259-openshift-default-fcb796d6c-9k9p9"
Jul 07 16:15:40.060379 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:15:40.060350    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-76e259-openshift-default-fcb796d6c-9k9p9"
Jul 07 16:15:40.800961 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:15:40.800927    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-76e259-openshift-default-fcb796d6c-9k9p9"
Jul 07 16:15:40.801857 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:15:40.801840    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-76e259-openshift-default-fcb796d6c-9k9p9"
Jul 07 16:15:40.923719 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:15:40.923680    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mk99s"]
Jul 07 16:15:50.140294 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:15:50.140258    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-be8b55/test-model-be8b55-kserve-67cf6f7675-s7w4l"]
Jul 07 16:15:50.143756 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:15:50.143738    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-be8b55/test-model-be8b55-kserve-67cf6f7675-s7w4l"
Jul 07 16:15:50.146919 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:15:50.146878    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-be8b55\"/\"openshift-service-ca.crt\""
Jul 07 16:15:50.147049 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:15:50.146912    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-be8b55\"/\"kube-root-ca.crt\""
Jul 07 16:15:50.147049 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:15:50.146914    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-be8b55\"/\"test-model-be8b55-kserve-self-signed-certs\""
Jul 07 16:15:50.147197 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:15:50.147181    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-be8b55\"/\"default-dockercfg-dtffb\""
Jul 07 16:15:50.153636 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:15:50.153608    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-be8b55/test-model-be8b55-kserve-67cf6f7675-s7w4l"]
Jul 07 16:15:50.243166 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:15:50.243134    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/092ccdee-e352-450e-848f-cc31ae63ffe2-model-cache\") pod \"test-model-be8b55-kserve-67cf6f7675-s7w4l\" (UID: \"092ccdee-e352-450e-848f-cc31ae63ffe2\") " pod="ai-tenant-e2e-inf-a-be8b55/test-model-be8b55-kserve-67cf6f7675-s7w4l"
Jul 07 16:15:50.243378 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:15:50.243186    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/092ccdee-e352-450e-848f-cc31ae63ffe2-tmp-dir\") pod \"test-model-be8b55-kserve-67cf6f7675-s7w4l\" (UID: \"092ccdee-e352-450e-848f-cc31ae63ffe2\") " pod="ai-tenant-e2e-inf-a-be8b55/test-model-be8b55-kserve-67cf6f7675-s7w4l"
Jul 07 16:15:50.243378 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:15:50.243213    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/092ccdee-e352-450e-848f-cc31ae63ffe2-dshm\") pod \"test-model-be8b55-kserve-67cf6f7675-s7w4l\" (UID: \"092ccdee-e352-450e-848f-cc31ae63ffe2\") " pod="ai-tenant-e2e-inf-a-be8b55/test-model-be8b55-kserve-67cf6f7675-s7w4l"
Jul 07 16:15:50.243378 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:15:50.243333    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/092ccdee-e352-450e-848f-cc31ae63ffe2-home\") pod \"test-model-be8b55-kserve-67cf6f7675-s7w4l\" (UID: \"092ccdee-e352-450e-848f-cc31ae63ffe2\") " pod="ai-tenant-e2e-inf-a-be8b55/test-model-be8b55-kserve-67cf6f7675-s7w4l"
Jul 07 16:15:50.243512 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:15:50.243375    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fkp7\" (UniqueName: \"kubernetes.io/projected/092ccdee-e352-450e-848f-cc31ae63ffe2-kube-api-access-5fkp7\") pod \"test-model-be8b55-kserve-67cf6f7675-s7w4l\" (UID: \"092ccdee-e352-450e-848f-cc31ae63ffe2\") " pod="ai-tenant-e2e-inf-a-be8b55/test-model-be8b55-kserve-67cf6f7675-s7w4l"
Jul 07 16:15:50.243512 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:15:50.243435    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/092ccdee-e352-450e-848f-cc31ae63ffe2-tls-certs\") pod \"test-model-be8b55-kserve-67cf6f7675-s7w4l\" (UID: \"092ccdee-e352-450e-848f-cc31ae63ffe2\") " pod="ai-tenant-e2e-inf-a-be8b55/test-model-be8b55-kserve-67cf6f7675-s7w4l"
Jul 07 16:15:50.344092 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:15:50.344056    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/092ccdee-e352-450e-848f-cc31ae63ffe2-dshm\") pod \"test-model-be8b55-kserve-67cf6f7675-s7w4l\" (UID: \"092ccdee-e352-450e-848f-cc31ae63ffe2\") " pod="ai-tenant-e2e-inf-a-be8b55/test-model-be8b55-kserve-67cf6f7675-s7w4l"
Jul 07 16:15:50.344304 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:15:50.344110    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/092ccdee-e352-450e-848f-cc31ae63ffe2-home\") pod \"test-model-be8b55-kserve-67cf6f7675-s7w4l\" (UID: \"092ccdee-e352-450e-848f-cc31ae63ffe2\") " pod="ai-tenant-e2e-inf-a-be8b55/test-model-be8b55-kserve-67cf6f7675-s7w4l"
Jul 07 16:15:50.344304 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:15:50.344236    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fkp7\" (UniqueName: \"kubernetes.io/projected/092ccdee-e352-450e-848f-cc31ae63ffe2-kube-api-access-5fkp7\") pod \"test-model-be8b55-kserve-67cf6f7675-s7w4l\" (UID: \"092ccdee-e352-450e-848f-cc31ae63ffe2\") " pod="ai-tenant-e2e-inf-a-be8b55/test-model-be8b55-kserve-67cf6f7675-s7w4l"
Jul 07 16:15:50.344304 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:15:50.344283    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/092ccdee-e352-450e-848f-cc31ae63ffe2-tls-certs\") pod \"test-model-be8b55-kserve-67cf6f7675-s7w4l\" (UID: \"092ccdee-e352-450e-848f-cc31ae63ffe2\") " pod="ai-tenant-e2e-inf-a-be8b55/test-model-be8b55-kserve-67cf6f7675-s7w4l"
Jul 07 16:15:50.344467 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:15:50.344330    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/092ccdee-e352-450e-848f-cc31ae63ffe2-model-cache\") pod \"test-model-be8b55-kserve-67cf6f7675-s7w4l\" (UID: \"092ccdee-e352-450e-848f-cc31ae63ffe2\") " pod="ai-tenant-e2e-inf-a-be8b55/test-model-be8b55-kserve-67cf6f7675-s7w4l"
Jul 07 16:15:50.344467 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:15:50.344403    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/092ccdee-e352-450e-848f-cc31ae63ffe2-tmp-dir\") pod \"test-model-be8b55-kserve-67cf6f7675-s7w4l\" (UID: \"092ccdee-e352-450e-848f-cc31ae63ffe2\") " pod="ai-tenant-e2e-inf-a-be8b55/test-model-be8b55-kserve-67cf6f7675-s7w4l"
Jul 07 16:15:50.344616 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:15:50.344501    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/092ccdee-e352-450e-848f-cc31ae63ffe2-home\") pod \"test-model-be8b55-kserve-67cf6f7675-s7w4l\" (UID: \"092ccdee-e352-450e-848f-cc31ae63ffe2\") " pod="ai-tenant-e2e-inf-a-be8b55/test-model-be8b55-kserve-67cf6f7675-s7w4l"
Jul 07 16:15:50.344680 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:15:50.344640    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/092ccdee-e352-450e-848f-cc31ae63ffe2-model-cache\") pod \"test-model-be8b55-kserve-67cf6f7675-s7w4l\" (UID: \"092ccdee-e352-450e-848f-cc31ae63ffe2\") " pod="ai-tenant-e2e-inf-a-be8b55/test-model-be8b55-kserve-67cf6f7675-s7w4l"
Jul 07 16:15:50.344782 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:15:50.344763    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/092ccdee-e352-450e-848f-cc31ae63ffe2-tmp-dir\") pod \"test-model-be8b55-kserve-67cf6f7675-s7w4l\" (UID: \"092ccdee-e352-450e-848f-cc31ae63ffe2\") " pod="ai-tenant-e2e-inf-a-be8b55/test-model-be8b55-kserve-67cf6f7675-s7w4l"
Jul 07 16:15:50.346401 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:15:50.346374    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/092ccdee-e352-450e-848f-cc31ae63ffe2-dshm\") pod \"test-model-be8b55-kserve-67cf6f7675-s7w4l\" (UID: \"092ccdee-e352-450e-848f-cc31ae63ffe2\") " pod="ai-tenant-e2e-inf-a-be8b55/test-model-be8b55-kserve-67cf6f7675-s7w4l"
Jul 07 16:15:50.346726 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:15:50.346709    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/092ccdee-e352-450e-848f-cc31ae63ffe2-tls-certs\") pod \"test-model-be8b55-kserve-67cf6f7675-s7w4l\" (UID: \"092ccdee-e352-450e-848f-cc31ae63ffe2\") " pod="ai-tenant-e2e-inf-a-be8b55/test-model-be8b55-kserve-67cf6f7675-s7w4l"
Jul 07 16:15:50.352908 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:15:50.352865    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fkp7\" (UniqueName: \"kubernetes.io/projected/092ccdee-e352-450e-848f-cc31ae63ffe2-kube-api-access-5fkp7\") pod \"test-model-be8b55-kserve-67cf6f7675-s7w4l\" (UID: \"092ccdee-e352-450e-848f-cc31ae63ffe2\") " pod="ai-tenant-e2e-inf-a-be8b55/test-model-be8b55-kserve-67cf6f7675-s7w4l"
Jul 07 16:15:50.454782 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:15:50.454750    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-be8b55/test-model-be8b55-kserve-67cf6f7675-s7w4l"
Jul 07 16:15:50.641874 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:15:50.641846    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-be8b55/test-model-be8b55-kserve-67cf6f7675-s7w4l"]
Jul 07 16:15:50.643998 ip-10-0-138-101 kubenswrapper[2566]: W0707 16:15:50.643965    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod092ccdee_e352_450e_848f_cc31ae63ffe2.slice/crio-fe353a88d3ed49ca9e08ee979e479de0592ab9899fb7186449e4160a31863ebe WatchSource:0}: Error finding container fe353a88d3ed49ca9e08ee979e479de0592ab9899fb7186449e4160a31863ebe: Status 404 returned error can't find the container with id fe353a88d3ed49ca9e08ee979e479de0592ab9899fb7186449e4160a31863ebe
Jul 07 16:15:50.847072 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:15:50.847028    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-be8b55/test-model-be8b55-kserve-67cf6f7675-s7w4l" event={"ID":"092ccdee-e352-450e-848f-cc31ae63ffe2","Type":"ContainerStarted","Data":"adedc9f75ebd8152f2fb8a5343fca9635ad14fa39f5ca7bd1993c0d4a8872cbc"}
Jul 07 16:15:50.847271 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:15:50.847083    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-be8b55/test-model-be8b55-kserve-67cf6f7675-s7w4l" event={"ID":"092ccdee-e352-450e-848f-cc31ae63ffe2","Type":"ContainerStarted","Data":"fe353a88d3ed49ca9e08ee979e479de0592ab9899fb7186449e4160a31863ebe"}
Jul 07 16:15:50.866673 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:15:50.866594    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-be8b55/test-model-be8b55-kserve-67cf6f7675-s7w4l" podStartSLOduration=0.86657089 podStartE2EDuration="866.57089ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:15:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:15:50.864006449 +0000 UTC m=+1904.952432123" watchObservedRunningTime="2026-07-07 16:15:50.86657089 +0000 UTC m=+1904.954996561"
Jul 07 16:15:54.546459 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:15:54.546422    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mk99s"]
Jul 07 16:15:55.802476 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:15:55.802441    2566 patch_prober.go:28] interesting pod/e2e-inf-b-76e259-openshift-default-fcb796d6c-9k9p9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Jul 07 16:15:55.802903 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:15:55.802504    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-76e259-openshift-default-fcb796d6c-9k9p9" podUID="cebc66f0-4d97-41bd-acb3-73ddbbd88260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Jul 07 16:15:55.870289 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:15:55.870250    2566 generic.go:356] "Generic (PLEG): container finished" podID="cebc66f0-4d97-41bd-acb3-73ddbbd88260" containerID="b33f8945a6a4c912602847a245a333e3ece59768806d0bdfee08cc2597c6b309" exitCode=0
Jul 07 16:15:55.870480 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:15:55.870309    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-76e259-openshift-default-fcb796d6c-9k9p9" event={"ID":"cebc66f0-4d97-41bd-acb3-73ddbbd88260","Type":"ContainerDied","Data":"b33f8945a6a4c912602847a245a333e3ece59768806d0bdfee08cc2597c6b309"}
Jul 07 16:15:55.870761 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:15:55.870745    2566 scope.go:117] "RemoveContainer" containerID="b33f8945a6a4c912602847a245a333e3ece59768806d0bdfee08cc2597c6b309"
Jul 07 16:15:55.871799 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:15:55.871768    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 07 16:15:55.871915 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:15:55.871860    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 07 16:15:55.872005 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:15:55.871924    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 07 16:15:56.889912 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:15:56.884773    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-76e259-openshift-default-fcb796d6c-9k9p9" event={"ID":"cebc66f0-4d97-41bd-acb3-73ddbbd88260","Type":"ContainerStarted","Data":"0c817cbc9dccd6a464abf20eaa3e207ed61ddc9f301b4508bb871e7bb111b6f1"}
Jul 07 16:15:57.055080 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:15:57.055042    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-76e259-openshift-default-fcb796d6c-9k9p9"
Jul 07 16:15:57.055413 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:15:57.055391    2566 patch_prober.go:28] interesting pod/e2e-inf-b-76e259-openshift-default-fcb796d6c-9k9p9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Jul 07 16:15:57.055472 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:15:57.055452    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-76e259-openshift-default-fcb796d6c-9k9p9" podUID="cebc66f0-4d97-41bd-acb3-73ddbbd88260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Jul 07 16:15:58.055731 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:15:58.055694    2566 patch_prober.go:28] interesting pod/e2e-inf-b-76e259-openshift-default-fcb796d6c-9k9p9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Jul 07 16:15:58.056164 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:15:58.055764    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-76e259-openshift-default-fcb796d6c-9k9p9" podUID="cebc66f0-4d97-41bd-acb3-73ddbbd88260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Jul 07 16:15:59.055728 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:15:59.055690    2566 patch_prober.go:28] interesting pod/e2e-inf-b-76e259-openshift-default-fcb796d6c-9k9p9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Jul 07 16:15:59.055937 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:15:59.055749    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-76e259-openshift-default-fcb796d6c-9k9p9" podUID="cebc66f0-4d97-41bd-acb3-73ddbbd88260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Jul 07 16:16:00.055501 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:16:00.055462    2566 patch_prober.go:28] interesting pod/e2e-inf-b-76e259-openshift-default-fcb796d6c-9k9p9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Jul 07 16:16:00.055685 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:16:00.055525    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-76e259-openshift-default-fcb796d6c-9k9p9" podUID="cebc66f0-4d97-41bd-acb3-73ddbbd88260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Jul 07 16:16:00.455874 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:16:00.455832    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-be8b55/test-model-be8b55-kserve-67cf6f7675-s7w4l"
Jul 07 16:16:00.456392 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:16:00.455914    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-be8b55/test-model-be8b55-kserve-67cf6f7675-s7w4l"
Jul 07 16:16:00.495104 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:16:00.495069    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-be8b55/test-model-be8b55-kserve-67cf6f7675-s7w4l"
Jul 07 16:16:00.917780 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:16:00.917702    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-be8b55/test-model-be8b55-kserve-67cf6f7675-s7w4l"
Jul 07 16:16:01.059203 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:16:01.059173    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-76e259-openshift-default-fcb796d6c-9k9p9"
Jul 07 16:16:01.059539 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:16:01.059520    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-76e259-openshift-default-fcb796d6c-9k9p9"
Jul 07 16:16:01.060140 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:16:01.060126    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-76e259-openshift-default-fcb796d6c-9k9p9"
Jul 07 16:16:09.720180 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:16:09.720148    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-76e259/test-model-76e259-kserve-6d49895ff7-kkvpn"]
Jul 07 16:16:09.723821 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:16:09.723798    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-76e259/test-model-76e259-kserve-6d49895ff7-kkvpn"
Jul 07 16:16:09.727166 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:16:09.727144    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-76e259\"/\"kube-root-ca.crt\""
Jul 07 16:16:09.727166 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:16:09.727165    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-76e259\"/\"openshift-service-ca.crt\""
Jul 07 16:16:09.727356 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:16:09.727147    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-76e259\"/\"default-dockercfg-ftpg9\""
Jul 07 16:16:09.727356 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:16:09.727146    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-76e259\"/\"test-model-76e259-kserve-self-signed-certs\""
Jul 07 16:16:09.734462 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:16:09.734438    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-76e259/test-model-76e259-kserve-6d49895ff7-kkvpn"]
Jul 07 16:16:09.824585 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:16:09.824541    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/250a5465-c446-46c3-8bca-d2bffc3e0139-home\") pod \"test-model-76e259-kserve-6d49895ff7-kkvpn\" (UID: \"250a5465-c446-46c3-8bca-d2bffc3e0139\") " pod="ai-tenant-e2e-inf-b-76e259/test-model-76e259-kserve-6d49895ff7-kkvpn"
Jul 07 16:16:09.824781 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:16:09.824600    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hf2hj\" (UniqueName: \"kubernetes.io/projected/250a5465-c446-46c3-8bca-d2bffc3e0139-kube-api-access-hf2hj\") pod \"test-model-76e259-kserve-6d49895ff7-kkvpn\" (UID: \"250a5465-c446-46c3-8bca-d2bffc3e0139\") " pod="ai-tenant-e2e-inf-b-76e259/test-model-76e259-kserve-6d49895ff7-kkvpn"
Jul 07 16:16:09.824781 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:16:09.824623    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/250a5465-c446-46c3-8bca-d2bffc3e0139-tls-certs\") pod \"test-model-76e259-kserve-6d49895ff7-kkvpn\" (UID: \"250a5465-c446-46c3-8bca-d2bffc3e0139\") " pod="ai-tenant-e2e-inf-b-76e259/test-model-76e259-kserve-6d49895ff7-kkvpn"
Jul 07 16:16:09.824781 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:16:09.824672    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/250a5465-c446-46c3-8bca-d2bffc3e0139-tmp-dir\") pod \"test-model-76e259-kserve-6d49895ff7-kkvpn\" (UID: \"250a5465-c446-46c3-8bca-d2bffc3e0139\") " pod="ai-tenant-e2e-inf-b-76e259/test-model-76e259-kserve-6d49895ff7-kkvpn"
Jul 07 16:16:09.824781 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:16:09.824747    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/250a5465-c446-46c3-8bca-d2bffc3e0139-model-cache\") pod \"test-model-76e259-kserve-6d49895ff7-kkvpn\" (UID: \"250a5465-c446-46c3-8bca-d2bffc3e0139\") " pod="ai-tenant-e2e-inf-b-76e259/test-model-76e259-kserve-6d49895ff7-kkvpn"
Jul 07 16:16:09.825005 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:16:09.824801    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/250a5465-c446-46c3-8bca-d2bffc3e0139-dshm\") pod \"test-model-76e259-kserve-6d49895ff7-kkvpn\" (UID: \"250a5465-c446-46c3-8bca-d2bffc3e0139\") " pod="ai-tenant-e2e-inf-b-76e259/test-model-76e259-kserve-6d49895ff7-kkvpn"
Jul 07 16:16:09.925492 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:16:09.925448    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/250a5465-c446-46c3-8bca-d2bffc3e0139-model-cache\") pod \"test-model-76e259-kserve-6d49895ff7-kkvpn\" (UID: \"250a5465-c446-46c3-8bca-d2bffc3e0139\") " pod="ai-tenant-e2e-inf-b-76e259/test-model-76e259-kserve-6d49895ff7-kkvpn"
Jul 07 16:16:09.925492 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:16:09.925495    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/250a5465-c446-46c3-8bca-d2bffc3e0139-dshm\") pod \"test-model-76e259-kserve-6d49895ff7-kkvpn\" (UID: \"250a5465-c446-46c3-8bca-d2bffc3e0139\") " pod="ai-tenant-e2e-inf-b-76e259/test-model-76e259-kserve-6d49895ff7-kkvpn"
Jul 07 16:16:09.925729 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:16:09.925539    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/250a5465-c446-46c3-8bca-d2bffc3e0139-home\") pod \"test-model-76e259-kserve-6d49895ff7-kkvpn\" (UID: \"250a5465-c446-46c3-8bca-d2bffc3e0139\") " pod="ai-tenant-e2e-inf-b-76e259/test-model-76e259-kserve-6d49895ff7-kkvpn"
Jul 07 16:16:09.925729 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:16:09.925573    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hf2hj\" (UniqueName: \"kubernetes.io/projected/250a5465-c446-46c3-8bca-d2bffc3e0139-kube-api-access-hf2hj\") pod \"test-model-76e259-kserve-6d49895ff7-kkvpn\" (UID: \"250a5465-c446-46c3-8bca-d2bffc3e0139\") " pod="ai-tenant-e2e-inf-b-76e259/test-model-76e259-kserve-6d49895ff7-kkvpn"
Jul 07 16:16:09.925729 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:16:09.925601    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/250a5465-c446-46c3-8bca-d2bffc3e0139-tls-certs\") pod \"test-model-76e259-kserve-6d49895ff7-kkvpn\" (UID: \"250a5465-c446-46c3-8bca-d2bffc3e0139\") " pod="ai-tenant-e2e-inf-b-76e259/test-model-76e259-kserve-6d49895ff7-kkvpn"
Jul 07 16:16:09.925729 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:16:09.925633    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/250a5465-c446-46c3-8bca-d2bffc3e0139-tmp-dir\") pod \"test-model-76e259-kserve-6d49895ff7-kkvpn\" (UID: \"250a5465-c446-46c3-8bca-d2bffc3e0139\") " pod="ai-tenant-e2e-inf-b-76e259/test-model-76e259-kserve-6d49895ff7-kkvpn"
Jul 07 16:16:09.925956 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:16:09.925928    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/250a5465-c446-46c3-8bca-d2bffc3e0139-model-cache\") pod \"test-model-76e259-kserve-6d49895ff7-kkvpn\" (UID: \"250a5465-c446-46c3-8bca-d2bffc3e0139\") " pod="ai-tenant-e2e-inf-b-76e259/test-model-76e259-kserve-6d49895ff7-kkvpn"
Jul 07 16:16:09.926006 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:16:09.925951    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/250a5465-c446-46c3-8bca-d2bffc3e0139-home\") pod \"test-model-76e259-kserve-6d49895ff7-kkvpn\" (UID: \"250a5465-c446-46c3-8bca-d2bffc3e0139\") " pod="ai-tenant-e2e-inf-b-76e259/test-model-76e259-kserve-6d49895ff7-kkvpn"
Jul 07 16:16:09.926065 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:16:09.926047    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/250a5465-c446-46c3-8bca-d2bffc3e0139-tmp-dir\") pod \"test-model-76e259-kserve-6d49895ff7-kkvpn\" (UID: \"250a5465-c446-46c3-8bca-d2bffc3e0139\") " pod="ai-tenant-e2e-inf-b-76e259/test-model-76e259-kserve-6d49895ff7-kkvpn"
Jul 07 16:16:09.928008 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:16:09.927984    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/250a5465-c446-46c3-8bca-d2bffc3e0139-dshm\") pod \"test-model-76e259-kserve-6d49895ff7-kkvpn\" (UID: \"250a5465-c446-46c3-8bca-d2bffc3e0139\") " pod="ai-tenant-e2e-inf-b-76e259/test-model-76e259-kserve-6d49895ff7-kkvpn"
Jul 07 16:16:09.928203 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:16:09.928186    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/250a5465-c446-46c3-8bca-d2bffc3e0139-tls-certs\") pod \"test-model-76e259-kserve-6d49895ff7-kkvpn\" (UID: \"250a5465-c446-46c3-8bca-d2bffc3e0139\") " pod="ai-tenant-e2e-inf-b-76e259/test-model-76e259-kserve-6d49895ff7-kkvpn"
Jul 07 16:16:09.933949 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:16:09.933867    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hf2hj\" (UniqueName: \"kubernetes.io/projected/250a5465-c446-46c3-8bca-d2bffc3e0139-kube-api-access-hf2hj\") pod \"test-model-76e259-kserve-6d49895ff7-kkvpn\" (UID: \"250a5465-c446-46c3-8bca-d2bffc3e0139\") " pod="ai-tenant-e2e-inf-b-76e259/test-model-76e259-kserve-6d49895ff7-kkvpn"
Jul 07 16:16:10.033327 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:16:10.033226    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-76e259/test-model-76e259-kserve-6d49895ff7-kkvpn"
Jul 07 16:16:10.178566 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:16:10.178437    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-76e259/test-model-76e259-kserve-6d49895ff7-kkvpn"]
Jul 07 16:16:10.181212 ip-10-0-138-101 kubenswrapper[2566]: W0707 16:16:10.181180    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod250a5465_c446_46c3_8bca_d2bffc3e0139.slice/crio-0c71656c79315a28b3520e64c31caabdc79c6f6bbc349aab64715ea911fdbb4c WatchSource:0}: Error finding container 0c71656c79315a28b3520e64c31caabdc79c6f6bbc349aab64715ea911fdbb4c: Status 404 returned error can't find the container with id 0c71656c79315a28b3520e64c31caabdc79c6f6bbc349aab64715ea911fdbb4c
Jul 07 16:16:10.955475 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:16:10.955436    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-76e259/test-model-76e259-kserve-6d49895ff7-kkvpn" event={"ID":"250a5465-c446-46c3-8bca-d2bffc3e0139","Type":"ContainerStarted","Data":"8eec2f2f0c58e53d11cc4259c6ff9de59664896e1234aa7ea4a1f5fb44595737"}
Jul 07 16:16:10.955878 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:16:10.955485    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-76e259/test-model-76e259-kserve-6d49895ff7-kkvpn" event={"ID":"250a5465-c446-46c3-8bca-d2bffc3e0139","Type":"ContainerStarted","Data":"0c71656c79315a28b3520e64c31caabdc79c6f6bbc349aab64715ea911fdbb4c"}
Jul 07 16:16:10.984753 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:16:10.984691    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-76e259/test-model-76e259-kserve-6d49895ff7-kkvpn" podStartSLOduration=1.984667594 podStartE2EDuration="1.984667594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:16:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:16:10.984278183 +0000 UTC m=+1925.072703867" watchObservedRunningTime="2026-07-07 16:16:10.984667594 +0000 UTC m=+1925.073093265"
Jul 07 16:16:16.430992 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:16:16.430957    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mk99s"]
Jul 07 16:16:20.034380 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:16:20.034287    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-76e259/test-model-76e259-kserve-6d49895ff7-kkvpn"
Jul 07 16:16:20.034765 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:16:20.034402    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-76e259/test-model-76e259-kserve-6d49895ff7-kkvpn"
Jul 07 16:16:20.046948 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:16:20.046923    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-76e259/test-model-76e259-kserve-6d49895ff7-kkvpn"
Jul 07 16:16:21.023971 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:16:21.023933    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-76e259/test-model-76e259-kserve-6d49895ff7-kkvpn"
Jul 07 16:16:34.687106 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:16:34.687072    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-be8b55/test-model-be8b55-kserve-67cf6f7675-s7w4l"]
Jul 07 16:16:35.588147 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:16:35.588109    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-76e259/test-model-76e259-kserve-6d49895ff7-kkvpn"]
Jul 07 16:16:39.443611 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:16:39.443576    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mk99s"]
Jul 07 16:16:49.706901 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:16:49.706830    2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-be8b55/test-model-be8b55-kserve-67cf6f7675-s7w4l" podUID="092ccdee-e352-450e-848f-cc31ae63ffe2" containerName="main" containerID="cri-o://adedc9f75ebd8152f2fb8a5343fca9635ad14fa39f5ca7bd1993c0d4a8872cbc" gracePeriod=45
Jul 07 16:16:49.956661 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:16:49.956636    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-be8b55/test-model-be8b55-kserve-67cf6f7675-s7w4l"
Jul 07 16:16:50.082668 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:16:50.082582    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/092ccdee-e352-450e-848f-cc31ae63ffe2-model-cache\") pod \"092ccdee-e352-450e-848f-cc31ae63ffe2\" (UID: \"092ccdee-e352-450e-848f-cc31ae63ffe2\") "
Jul 07 16:16:50.082668 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:16:50.082662    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/092ccdee-e352-450e-848f-cc31ae63ffe2-tls-certs\") pod \"092ccdee-e352-450e-848f-cc31ae63ffe2\" (UID: \"092ccdee-e352-450e-848f-cc31ae63ffe2\") "
Jul 07 16:16:50.082926 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:16:50.082691    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/092ccdee-e352-450e-848f-cc31ae63ffe2-home\") pod \"092ccdee-e352-450e-848f-cc31ae63ffe2\" (UID: \"092ccdee-e352-450e-848f-cc31ae63ffe2\") "
Jul 07 16:16:50.082926 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:16:50.082733    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/092ccdee-e352-450e-848f-cc31ae63ffe2-dshm\") pod \"092ccdee-e352-450e-848f-cc31ae63ffe2\" (UID: \"092ccdee-e352-450e-848f-cc31ae63ffe2\") "
Jul 07 16:16:50.082926 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:16:50.082757    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/092ccdee-e352-450e-848f-cc31ae63ffe2-tmp-dir\") pod \"092ccdee-e352-450e-848f-cc31ae63ffe2\" (UID: \"092ccdee-e352-450e-848f-cc31ae63ffe2\") "
Jul 07 16:16:50.082926 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:16:50.082777    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5fkp7\" (UniqueName: \"kubernetes.io/projected/092ccdee-e352-450e-848f-cc31ae63ffe2-kube-api-access-5fkp7\") pod \"092ccdee-e352-450e-848f-cc31ae63ffe2\" (UID: \"092ccdee-e352-450e-848f-cc31ae63ffe2\") "
Jul 07 16:16:50.083128 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:16:50.082960    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/092ccdee-e352-450e-848f-cc31ae63ffe2-home" (OuterVolumeSpecName: "home") pod "092ccdee-e352-450e-848f-cc31ae63ffe2" (UID: "092ccdee-e352-450e-848f-cc31ae63ffe2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 16:16:50.083128 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:16:50.082972    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/092ccdee-e352-450e-848f-cc31ae63ffe2-model-cache" (OuterVolumeSpecName: "model-cache") pod "092ccdee-e352-450e-848f-cc31ae63ffe2" (UID: "092ccdee-e352-450e-848f-cc31ae63ffe2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 16:16:50.083128 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:16:50.083058    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/092ccdee-e352-450e-848f-cc31ae63ffe2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "092ccdee-e352-450e-848f-cc31ae63ffe2" (UID: "092ccdee-e352-450e-848f-cc31ae63ffe2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 16:16:50.083267 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:16:50.083127    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/092ccdee-e352-450e-848f-cc31ae63ffe2-home\") on node \"ip-10-0-138-101.ec2.internal\" DevicePath \"\""
Jul 07 16:16:50.083267 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:16:50.083145    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/092ccdee-e352-450e-848f-cc31ae63ffe2-model-cache\") on node \"ip-10-0-138-101.ec2.internal\" DevicePath \"\""
Jul 07 16:16:50.085008 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:16:50.084983    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/092ccdee-e352-450e-848f-cc31ae63ffe2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "092ccdee-e352-450e-848f-cc31ae63ffe2" (UID: "092ccdee-e352-450e-848f-cc31ae63ffe2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 16:16:50.085085 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:16:50.085002    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/092ccdee-e352-450e-848f-cc31ae63ffe2-dshm" (OuterVolumeSpecName: "dshm") pod "092ccdee-e352-450e-848f-cc31ae63ffe2" (UID: "092ccdee-e352-450e-848f-cc31ae63ffe2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 16:16:50.085085 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:16:50.085067    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/092ccdee-e352-450e-848f-cc31ae63ffe2-kube-api-access-5fkp7" (OuterVolumeSpecName: "kube-api-access-5fkp7") pod "092ccdee-e352-450e-848f-cc31ae63ffe2" (UID: "092ccdee-e352-450e-848f-cc31ae63ffe2"). InnerVolumeSpecName "kube-api-access-5fkp7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 16:16:50.145563 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:16:50.145531    2566 generic.go:356] "Generic (PLEG): container finished" podID="092ccdee-e352-450e-848f-cc31ae63ffe2" containerID="adedc9f75ebd8152f2fb8a5343fca9635ad14fa39f5ca7bd1993c0d4a8872cbc" exitCode=0
Jul 07 16:16:50.145720 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:16:50.145567    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-be8b55/test-model-be8b55-kserve-67cf6f7675-s7w4l" event={"ID":"092ccdee-e352-450e-848f-cc31ae63ffe2","Type":"ContainerDied","Data":"adedc9f75ebd8152f2fb8a5343fca9635ad14fa39f5ca7bd1993c0d4a8872cbc"}
Jul 07 16:16:50.145720 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:16:50.145575    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-be8b55/test-model-be8b55-kserve-67cf6f7675-s7w4l"
Jul 07 16:16:50.145720 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:16:50.145594    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-be8b55/test-model-be8b55-kserve-67cf6f7675-s7w4l" event={"ID":"092ccdee-e352-450e-848f-cc31ae63ffe2","Type":"ContainerDied","Data":"fe353a88d3ed49ca9e08ee979e479de0592ab9899fb7186449e4160a31863ebe"}
Jul 07 16:16:50.145720 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:16:50.145611    2566 scope.go:117] "RemoveContainer" containerID="adedc9f75ebd8152f2fb8a5343fca9635ad14fa39f5ca7bd1993c0d4a8872cbc"
Jul 07 16:16:50.160038 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:16:50.160014    2566 scope.go:117] "RemoveContainer" containerID="adedc9f75ebd8152f2fb8a5343fca9635ad14fa39f5ca7bd1993c0d4a8872cbc"
Jul 07 16:16:50.160376 ip-10-0-138-101 kubenswrapper[2566]: E0707 16:16:50.160354    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adedc9f75ebd8152f2fb8a5343fca9635ad14fa39f5ca7bd1993c0d4a8872cbc\": container with ID starting with adedc9f75ebd8152f2fb8a5343fca9635ad14fa39f5ca7bd1993c0d4a8872cbc not found: ID does not exist" containerID="adedc9f75ebd8152f2fb8a5343fca9635ad14fa39f5ca7bd1993c0d4a8872cbc"
Jul 07 16:16:50.160459 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:16:50.160385    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adedc9f75ebd8152f2fb8a5343fca9635ad14fa39f5ca7bd1993c0d4a8872cbc"} err="failed to get container status \"adedc9f75ebd8152f2fb8a5343fca9635ad14fa39f5ca7bd1993c0d4a8872cbc\": rpc error: code = NotFound desc = could not find container \"adedc9f75ebd8152f2fb8a5343fca9635ad14fa39f5ca7bd1993c0d4a8872cbc\": container with ID starting with adedc9f75ebd8152f2fb8a5343fca9635ad14fa39f5ca7bd1993c0d4a8872cbc not found: ID does not exist"
Jul 07 16:16:50.173066 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:16:50.173031    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-be8b55/test-model-be8b55-kserve-67cf6f7675-s7w4l"]
Jul 07 16:16:50.179050 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:16:50.179021    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-be8b55/test-model-be8b55-kserve-67cf6f7675-s7w4l"]
Jul 07 16:16:50.184500 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:16:50.184477    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/092ccdee-e352-450e-848f-cc31ae63ffe2-tls-certs\") on node \"ip-10-0-138-101.ec2.internal\" DevicePath \"\""
Jul 07 16:16:50.184570 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:16:50.184502    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/092ccdee-e352-450e-848f-cc31ae63ffe2-dshm\") on node \"ip-10-0-138-101.ec2.internal\" DevicePath \"\""
Jul 07 16:16:50.184570 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:16:50.184511    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/092ccdee-e352-450e-848f-cc31ae63ffe2-tmp-dir\") on node \"ip-10-0-138-101.ec2.internal\" DevicePath \"\""
Jul 07 16:16:50.184570 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:16:50.184520    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fkp7\" (UniqueName: \"kubernetes.io/projected/092ccdee-e352-450e-848f-cc31ae63ffe2-kube-api-access-5fkp7\") on node \"ip-10-0-138-101.ec2.internal\" DevicePath \"\""
Jul 07 16:16:50.475301 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:16:50.475268    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="092ccdee-e352-450e-848f-cc31ae63ffe2" path="/var/lib/kubelet/pods/092ccdee-e352-450e-848f-cc31ae63ffe2/volumes"
Jul 07 16:16:50.609649 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:16:50.609586    2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-76e259/test-model-76e259-kserve-6d49895ff7-kkvpn" podUID="250a5465-c446-46c3-8bca-d2bffc3e0139" containerName="main" containerID="cri-o://8eec2f2f0c58e53d11cc4259c6ff9de59664896e1234aa7ea4a1f5fb44595737" gracePeriod=45
Jul 07 16:16:50.857455 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:16:50.857426    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-76e259/test-model-76e259-kserve-6d49895ff7-kkvpn"
Jul 07 16:16:50.992229 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:16:50.992130    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hf2hj\" (UniqueName: \"kubernetes.io/projected/250a5465-c446-46c3-8bca-d2bffc3e0139-kube-api-access-hf2hj\") pod \"250a5465-c446-46c3-8bca-d2bffc3e0139\" (UID: \"250a5465-c446-46c3-8bca-d2bffc3e0139\") "
Jul 07 16:16:50.992229 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:16:50.992187    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/250a5465-c446-46c3-8bca-d2bffc3e0139-model-cache\") pod \"250a5465-c446-46c3-8bca-d2bffc3e0139\" (UID: \"250a5465-c446-46c3-8bca-d2bffc3e0139\") "
Jul 07 16:16:50.992229 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:16:50.992234    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/250a5465-c446-46c3-8bca-d2bffc3e0139-dshm\") pod \"250a5465-c446-46c3-8bca-d2bffc3e0139\" (UID: \"250a5465-c446-46c3-8bca-d2bffc3e0139\") "
Jul 07 16:16:50.992446 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:16:50.992392    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/250a5465-c446-46c3-8bca-d2bffc3e0139-tmp-dir\") pod \"250a5465-c446-46c3-8bca-d2bffc3e0139\" (UID: \"250a5465-c446-46c3-8bca-d2bffc3e0139\") "
Jul 07 16:16:50.992446 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:16:50.992442    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/250a5465-c446-46c3-8bca-d2bffc3e0139-home\") pod \"250a5465-c446-46c3-8bca-d2bffc3e0139\" (UID: \"250a5465-c446-46c3-8bca-d2bffc3e0139\") "
Jul 07 16:16:50.992519 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:16:50.992491    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/250a5465-c446-46c3-8bca-d2bffc3e0139-model-cache" (OuterVolumeSpecName: "model-cache") pod "250a5465-c446-46c3-8bca-d2bffc3e0139" (UID: "250a5465-c446-46c3-8bca-d2bffc3e0139"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 16:16:50.992570 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:16:50.992505    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/250a5465-c446-46c3-8bca-d2bffc3e0139-tls-certs\") pod \"250a5465-c446-46c3-8bca-d2bffc3e0139\" (UID: \"250a5465-c446-46c3-8bca-d2bffc3e0139\") "
Jul 07 16:16:50.992680 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:16:50.992635    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/250a5465-c446-46c3-8bca-d2bffc3e0139-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "250a5465-c446-46c3-8bca-d2bffc3e0139" (UID: "250a5465-c446-46c3-8bca-d2bffc3e0139"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 16:16:50.992804 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:16:50.992718    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/250a5465-c446-46c3-8bca-d2bffc3e0139-home" (OuterVolumeSpecName: "home") pod "250a5465-c446-46c3-8bca-d2bffc3e0139" (UID: "250a5465-c446-46c3-8bca-d2bffc3e0139"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 16:16:50.992849 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:16:50.992826    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/250a5465-c446-46c3-8bca-d2bffc3e0139-tmp-dir\") on node \"ip-10-0-138-101.ec2.internal\" DevicePath \"\""
Jul 07 16:16:50.992849 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:16:50.992840    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/250a5465-c446-46c3-8bca-d2bffc3e0139-home\") on node \"ip-10-0-138-101.ec2.internal\" DevicePath \"\""
Jul 07 16:16:50.992849 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:16:50.992849    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/250a5465-c446-46c3-8bca-d2bffc3e0139-model-cache\") on node \"ip-10-0-138-101.ec2.internal\" DevicePath \"\""
Jul 07 16:16:50.994592 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:16:50.994571    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/250a5465-c446-46c3-8bca-d2bffc3e0139-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "250a5465-c446-46c3-8bca-d2bffc3e0139" (UID: "250a5465-c446-46c3-8bca-d2bffc3e0139"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 16:16:50.994824 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:16:50.994796    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/250a5465-c446-46c3-8bca-d2bffc3e0139-dshm" (OuterVolumeSpecName: "dshm") pod "250a5465-c446-46c3-8bca-d2bffc3e0139" (UID: "250a5465-c446-46c3-8bca-d2bffc3e0139"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 16:16:50.994824 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:16:50.994817    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/250a5465-c446-46c3-8bca-d2bffc3e0139-kube-api-access-hf2hj" (OuterVolumeSpecName: "kube-api-access-hf2hj") pod "250a5465-c446-46c3-8bca-d2bffc3e0139" (UID: "250a5465-c446-46c3-8bca-d2bffc3e0139"). InnerVolumeSpecName "kube-api-access-hf2hj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 16:16:51.093310 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:16:51.093270    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/250a5465-c446-46c3-8bca-d2bffc3e0139-tls-certs\") on node \"ip-10-0-138-101.ec2.internal\" DevicePath \"\""
Jul 07 16:16:51.093310 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:16:51.093305    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hf2hj\" (UniqueName: \"kubernetes.io/projected/250a5465-c446-46c3-8bca-d2bffc3e0139-kube-api-access-hf2hj\") on node \"ip-10-0-138-101.ec2.internal\" DevicePath \"\""
Jul 07 16:16:51.093310 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:16:51.093316    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/250a5465-c446-46c3-8bca-d2bffc3e0139-dshm\") on node \"ip-10-0-138-101.ec2.internal\" DevicePath \"\""
Jul 07 16:16:51.152140 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:16:51.152110    2566 generic.go:356] "Generic (PLEG): container finished" podID="250a5465-c446-46c3-8bca-d2bffc3e0139" containerID="8eec2f2f0c58e53d11cc4259c6ff9de59664896e1234aa7ea4a1f5fb44595737" exitCode=0
Jul 07 16:16:51.152293 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:16:51.152149    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-76e259/test-model-76e259-kserve-6d49895ff7-kkvpn"
Jul 07 16:16:51.152293 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:16:51.152186    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-76e259/test-model-76e259-kserve-6d49895ff7-kkvpn" event={"ID":"250a5465-c446-46c3-8bca-d2bffc3e0139","Type":"ContainerDied","Data":"8eec2f2f0c58e53d11cc4259c6ff9de59664896e1234aa7ea4a1f5fb44595737"}
Jul 07 16:16:51.152293 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:16:51.152226    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-76e259/test-model-76e259-kserve-6d49895ff7-kkvpn" event={"ID":"250a5465-c446-46c3-8bca-d2bffc3e0139","Type":"ContainerDied","Data":"0c71656c79315a28b3520e64c31caabdc79c6f6bbc349aab64715ea911fdbb4c"}
Jul 07 16:16:51.152293 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:16:51.152248    2566 scope.go:117] "RemoveContainer" containerID="8eec2f2f0c58e53d11cc4259c6ff9de59664896e1234aa7ea4a1f5fb44595737"
Jul 07 16:16:51.161119 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:16:51.161096    2566 scope.go:117] "RemoveContainer" containerID="8eec2f2f0c58e53d11cc4259c6ff9de59664896e1234aa7ea4a1f5fb44595737"
Jul 07 16:16:51.161462 ip-10-0-138-101 kubenswrapper[2566]: E0707 16:16:51.161436    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8eec2f2f0c58e53d11cc4259c6ff9de59664896e1234aa7ea4a1f5fb44595737\": container with ID starting with 8eec2f2f0c58e53d11cc4259c6ff9de59664896e1234aa7ea4a1f5fb44595737 not found: ID does not exist" containerID="8eec2f2f0c58e53d11cc4259c6ff9de59664896e1234aa7ea4a1f5fb44595737"
Jul 07 16:16:51.161558 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:16:51.161475    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8eec2f2f0c58e53d11cc4259c6ff9de59664896e1234aa7ea4a1f5fb44595737"} err="failed to get container status \"8eec2f2f0c58e53d11cc4259c6ff9de59664896e1234aa7ea4a1f5fb44595737\": rpc error: code = NotFound desc = could not find container \"8eec2f2f0c58e53d11cc4259c6ff9de59664896e1234aa7ea4a1f5fb44595737\": container with ID starting with 8eec2f2f0c58e53d11cc4259c6ff9de59664896e1234aa7ea4a1f5fb44595737 not found: ID does not exist"
Jul 07 16:16:51.179557 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:16:51.179527    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-76e259/test-model-76e259-kserve-6d49895ff7-kkvpn"]
Jul 07 16:16:51.185253 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:16:51.185226    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-76e259/test-model-76e259-kserve-6d49895ff7-kkvpn"]
Jul 07 16:16:52.474795 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:16:52.474761    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="250a5465-c446-46c3-8bca-d2bffc3e0139" path="/var/lib/kubelet/pods/250a5465-c446-46c3-8bca-d2bffc3e0139/volumes"
Jul 07 16:17:23.462125 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:23.462094    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mk99s"]
Jul 07 16:17:28.817594 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:28.817559    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-76e259-openshift-default-fcb796d6c-9k9p9"]
Jul 07 16:17:28.818131 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:28.817769    2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-76e259-openshift-default-fcb796d6c-9k9p9" podUID="cebc66f0-4d97-41bd-acb3-73ddbbd88260" containerName="istio-proxy" containerID="cri-o://0c817cbc9dccd6a464abf20eaa3e207ed61ddc9f301b4508bb871e7bb111b6f1" gracePeriod=30
Jul 07 16:17:31.061948 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:31.061909    2566 patch_prober.go:28] interesting pod/e2e-inf-b-76e259-openshift-default-fcb796d6c-9k9p9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 07 16:17:31.062323 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:31.061966    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-76e259-openshift-default-fcb796d6c-9k9p9" podUID="cebc66f0-4d97-41bd-acb3-73ddbbd88260" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 16:17:34.076587 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:34.076517    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-76e259-openshift-default-fcb796d6c-9k9p9"
Jul 07 16:17:34.157537 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:34.157504    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cebc66f0-4d97-41bd-acb3-73ddbbd88260-istio-token\") pod \"cebc66f0-4d97-41bd-acb3-73ddbbd88260\" (UID: \"cebc66f0-4d97-41bd-acb3-73ddbbd88260\") "
Jul 07 16:17:34.157715 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:34.157571    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cebc66f0-4d97-41bd-acb3-73ddbbd88260-istio-data\") pod \"cebc66f0-4d97-41bd-acb3-73ddbbd88260\" (UID: \"cebc66f0-4d97-41bd-acb3-73ddbbd88260\") "
Jul 07 16:17:34.157715 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:34.157672    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cebc66f0-4d97-41bd-acb3-73ddbbd88260-istiod-ca-cert\") pod \"cebc66f0-4d97-41bd-acb3-73ddbbd88260\" (UID: \"cebc66f0-4d97-41bd-acb3-73ddbbd88260\") "
Jul 07 16:17:34.157715 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:34.157708    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cebc66f0-4d97-41bd-acb3-73ddbbd88260-workload-certs\") pod \"cebc66f0-4d97-41bd-acb3-73ddbbd88260\" (UID: \"cebc66f0-4d97-41bd-acb3-73ddbbd88260\") "
Jul 07 16:17:34.157879 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:34.157778    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cebc66f0-4d97-41bd-acb3-73ddbbd88260-credential-socket\") pod \"cebc66f0-4d97-41bd-acb3-73ddbbd88260\" (UID: \"cebc66f0-4d97-41bd-acb3-73ddbbd88260\") "
Jul 07 16:17:34.157879 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:34.157842    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cebc66f0-4d97-41bd-acb3-73ddbbd88260-istio-podinfo\") pod \"cebc66f0-4d97-41bd-acb3-73ddbbd88260\" (UID: \"cebc66f0-4d97-41bd-acb3-73ddbbd88260\") "
Jul 07 16:17:34.158007 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:34.157904    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cebc66f0-4d97-41bd-acb3-73ddbbd88260-workload-socket\") pod \"cebc66f0-4d97-41bd-acb3-73ddbbd88260\" (UID: \"cebc66f0-4d97-41bd-acb3-73ddbbd88260\") "
Jul 07 16:17:34.158007 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:34.157935    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cebc66f0-4d97-41bd-acb3-73ddbbd88260-istio-envoy\") pod \"cebc66f0-4d97-41bd-acb3-73ddbbd88260\" (UID: \"cebc66f0-4d97-41bd-acb3-73ddbbd88260\") "
Jul 07 16:17:34.158007 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:34.157962    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7r5kb\" (UniqueName: \"kubernetes.io/projected/cebc66f0-4d97-41bd-acb3-73ddbbd88260-kube-api-access-7r5kb\") pod \"cebc66f0-4d97-41bd-acb3-73ddbbd88260\" (UID: \"cebc66f0-4d97-41bd-acb3-73ddbbd88260\") "
Jul 07 16:17:34.158146 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:34.158001    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cebc66f0-4d97-41bd-acb3-73ddbbd88260-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "cebc66f0-4d97-41bd-acb3-73ddbbd88260" (UID: "cebc66f0-4d97-41bd-acb3-73ddbbd88260"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 16:17:34.158146 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:34.158016    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cebc66f0-4d97-41bd-acb3-73ddbbd88260-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "cebc66f0-4d97-41bd-acb3-73ddbbd88260" (UID: "cebc66f0-4d97-41bd-acb3-73ddbbd88260"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 16:17:34.158146 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:34.158043    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cebc66f0-4d97-41bd-acb3-73ddbbd88260-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "cebc66f0-4d97-41bd-acb3-73ddbbd88260" (UID: "cebc66f0-4d97-41bd-acb3-73ddbbd88260"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 16:17:34.158292 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:34.158207    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cebc66f0-4d97-41bd-acb3-73ddbbd88260-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "cebc66f0-4d97-41bd-acb3-73ddbbd88260" (UID: "cebc66f0-4d97-41bd-acb3-73ddbbd88260"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 16:17:34.158292 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:34.158237    2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cebc66f0-4d97-41bd-acb3-73ddbbd88260-istiod-ca-cert\") on node \"ip-10-0-138-101.ec2.internal\" DevicePath \"\""
Jul 07 16:17:34.158292 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:34.158254    2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cebc66f0-4d97-41bd-acb3-73ddbbd88260-workload-certs\") on node \"ip-10-0-138-101.ec2.internal\" DevicePath \"\""
Jul 07 16:17:34.158292 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:34.158268    2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cebc66f0-4d97-41bd-acb3-73ddbbd88260-credential-socket\") on node \"ip-10-0-138-101.ec2.internal\" DevicePath \"\""
Jul 07 16:17:34.158503 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:34.158423    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cebc66f0-4d97-41bd-acb3-73ddbbd88260-istio-data" (OuterVolumeSpecName: "istio-data") pod "cebc66f0-4d97-41bd-acb3-73ddbbd88260" (UID: "cebc66f0-4d97-41bd-acb3-73ddbbd88260"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 16:17:34.160230 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:34.160198    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cebc66f0-4d97-41bd-acb3-73ddbbd88260-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "cebc66f0-4d97-41bd-acb3-73ddbbd88260" (UID: "cebc66f0-4d97-41bd-acb3-73ddbbd88260"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 16:17:34.160357 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:34.160271    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cebc66f0-4d97-41bd-acb3-73ddbbd88260-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "cebc66f0-4d97-41bd-acb3-73ddbbd88260" (UID: "cebc66f0-4d97-41bd-acb3-73ddbbd88260"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 07 16:17:34.160494 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:34.160471    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cebc66f0-4d97-41bd-acb3-73ddbbd88260-kube-api-access-7r5kb" (OuterVolumeSpecName: "kube-api-access-7r5kb") pod "cebc66f0-4d97-41bd-acb3-73ddbbd88260" (UID: "cebc66f0-4d97-41bd-acb3-73ddbbd88260"). InnerVolumeSpecName "kube-api-access-7r5kb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 16:17:34.160679 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:34.160662    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cebc66f0-4d97-41bd-acb3-73ddbbd88260-istio-token" (OuterVolumeSpecName: "istio-token") pod "cebc66f0-4d97-41bd-acb3-73ddbbd88260" (UID: "cebc66f0-4d97-41bd-acb3-73ddbbd88260"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 16:17:34.259176 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:34.259140    2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cebc66f0-4d97-41bd-acb3-73ddbbd88260-istio-podinfo\") on node \"ip-10-0-138-101.ec2.internal\" DevicePath \"\""
Jul 07 16:17:34.259176 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:34.259168    2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cebc66f0-4d97-41bd-acb3-73ddbbd88260-workload-socket\") on node \"ip-10-0-138-101.ec2.internal\" DevicePath \"\""
Jul 07 16:17:34.259176 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:34.259181    2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cebc66f0-4d97-41bd-acb3-73ddbbd88260-istio-envoy\") on node \"ip-10-0-138-101.ec2.internal\" DevicePath \"\""
Jul 07 16:17:34.259415 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:34.259195    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7r5kb\" (UniqueName: \"kubernetes.io/projected/cebc66f0-4d97-41bd-acb3-73ddbbd88260-kube-api-access-7r5kb\") on node \"ip-10-0-138-101.ec2.internal\" DevicePath \"\""
Jul 07 16:17:34.259415 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:34.259206    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cebc66f0-4d97-41bd-acb3-73ddbbd88260-istio-token\") on node \"ip-10-0-138-101.ec2.internal\" DevicePath \"\""
Jul 07 16:17:34.259415 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:34.259217    2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cebc66f0-4d97-41bd-acb3-73ddbbd88260-istio-data\") on node \"ip-10-0-138-101.ec2.internal\" DevicePath \"\""
Jul 07 16:17:34.365778 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:34.365675    2566 generic.go:356] "Generic (PLEG): container finished" podID="cebc66f0-4d97-41bd-acb3-73ddbbd88260" containerID="0c817cbc9dccd6a464abf20eaa3e207ed61ddc9f301b4508bb871e7bb111b6f1" exitCode=0
Jul 07 16:17:34.365778 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:34.365730    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-76e259-openshift-default-fcb796d6c-9k9p9"
Jul 07 16:17:34.365778 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:34.365759    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-76e259-openshift-default-fcb796d6c-9k9p9" event={"ID":"cebc66f0-4d97-41bd-acb3-73ddbbd88260","Type":"ContainerDied","Data":"0c817cbc9dccd6a464abf20eaa3e207ed61ddc9f301b4508bb871e7bb111b6f1"}
Jul 07 16:17:34.366074 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:34.365822    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-76e259-openshift-default-fcb796d6c-9k9p9" event={"ID":"cebc66f0-4d97-41bd-acb3-73ddbbd88260","Type":"ContainerDied","Data":"b07c6e3e1a7f18bb7d444d3e26c10b187669e1b3cbdc5bdc5514ecc474a42ade"}
Jul 07 16:17:34.366074 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:34.365859    2566 scope.go:117] "RemoveContainer" containerID="0c817cbc9dccd6a464abf20eaa3e207ed61ddc9f301b4508bb871e7bb111b6f1"
Jul 07 16:17:34.376630 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:34.376613    2566 scope.go:117] "RemoveContainer" containerID="b33f8945a6a4c912602847a245a333e3ece59768806d0bdfee08cc2597c6b309"
Jul 07 16:17:34.384606 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:34.384587    2566 scope.go:117] "RemoveContainer" containerID="0c817cbc9dccd6a464abf20eaa3e207ed61ddc9f301b4508bb871e7bb111b6f1"
Jul 07 16:17:34.384881 ip-10-0-138-101 kubenswrapper[2566]: E0707 16:17:34.384863    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c817cbc9dccd6a464abf20eaa3e207ed61ddc9f301b4508bb871e7bb111b6f1\": container with ID starting with 0c817cbc9dccd6a464abf20eaa3e207ed61ddc9f301b4508bb871e7bb111b6f1 not found: ID does not exist" containerID="0c817cbc9dccd6a464abf20eaa3e207ed61ddc9f301b4508bb871e7bb111b6f1"
Jul 07 16:17:34.385001 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:34.384905    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c817cbc9dccd6a464abf20eaa3e207ed61ddc9f301b4508bb871e7bb111b6f1"} err="failed to get container status \"0c817cbc9dccd6a464abf20eaa3e207ed61ddc9f301b4508bb871e7bb111b6f1\": rpc error: code = NotFound desc = could not find container \"0c817cbc9dccd6a464abf20eaa3e207ed61ddc9f301b4508bb871e7bb111b6f1\": container with ID starting with 0c817cbc9dccd6a464abf20eaa3e207ed61ddc9f301b4508bb871e7bb111b6f1 not found: ID does not exist"
Jul 07 16:17:34.385001 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:34.384922    2566 scope.go:117] "RemoveContainer" containerID="b33f8945a6a4c912602847a245a333e3ece59768806d0bdfee08cc2597c6b309"
Jul 07 16:17:34.385179 ip-10-0-138-101 kubenswrapper[2566]: E0707 16:17:34.385161    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b33f8945a6a4c912602847a245a333e3ece59768806d0bdfee08cc2597c6b309\": container with ID starting with b33f8945a6a4c912602847a245a333e3ece59768806d0bdfee08cc2597c6b309 not found: ID does not exist" containerID="b33f8945a6a4c912602847a245a333e3ece59768806d0bdfee08cc2597c6b309"
Jul 07 16:17:34.385222 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:34.385186    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b33f8945a6a4c912602847a245a333e3ece59768806d0bdfee08cc2597c6b309"} err="failed to get container status \"b33f8945a6a4c912602847a245a333e3ece59768806d0bdfee08cc2597c6b309\": rpc error: code = NotFound desc = could not find container \"b33f8945a6a4c912602847a245a333e3ece59768806d0bdfee08cc2597c6b309\": container with ID starting with b33f8945a6a4c912602847a245a333e3ece59768806d0bdfee08cc2597c6b309 not found: ID does not exist"
Jul 07 16:17:34.399364 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:34.399336    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-76e259-openshift-default-fcb796d6c-9k9p9"]
Jul 07 16:17:34.406879 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:34.406857    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-76e259-openshift-default-fcb796d6c-9k9p9"]
Jul 07 16:17:34.474452 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:34.474420    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cebc66f0-4d97-41bd-acb3-73ddbbd88260" path="/var/lib/kubelet/pods/cebc66f0-4d97-41bd-acb3-73ddbbd88260/volumes"
Jul 07 16:17:38.704092 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:38.704047    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-e60798-openshift-default-64cdfcbbff-t9qqp"]
Jul 07 16:17:38.704496 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:38.704384    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cebc66f0-4d97-41bd-acb3-73ddbbd88260" containerName="istio-proxy"
Jul 07 16:17:38.704496 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:38.704395    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="cebc66f0-4d97-41bd-acb3-73ddbbd88260" containerName="istio-proxy"
Jul 07 16:17:38.704496 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:38.704424    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="092ccdee-e352-450e-848f-cc31ae63ffe2" containerName="main"
Jul 07 16:17:38.704496 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:38.704430    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="092ccdee-e352-450e-848f-cc31ae63ffe2" containerName="main"
Jul 07 16:17:38.704496 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:38.704437    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="250a5465-c446-46c3-8bca-d2bffc3e0139" containerName="main"
Jul 07 16:17:38.704496 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:38.704443    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="250a5465-c446-46c3-8bca-d2bffc3e0139" containerName="main"
Jul 07 16:17:38.704816 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:38.704519    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="cebc66f0-4d97-41bd-acb3-73ddbbd88260" containerName="istio-proxy"
Jul 07 16:17:38.704816 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:38.704531    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="092ccdee-e352-450e-848f-cc31ae63ffe2" containerName="main"
Jul 07 16:17:38.704816 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:38.704546    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="250a5465-c446-46c3-8bca-d2bffc3e0139" containerName="main"
Jul 07 16:17:38.709375 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:38.709359    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e60798-openshift-default-64cdfcbbff-t9qqp"
Jul 07 16:17:38.712864 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:38.712846    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-e60798-openshift-default-dockercfg-m52lq\""
Jul 07 16:17:38.729895 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:38.729862    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-e60798-openshift-default-64cdfcbbff-t9qqp"]
Jul 07 16:17:38.800508 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:38.800474    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62a849d6-190a-4950-953c-565117367f4d-credential-socket\") pod \"e2e-api-b-e60798-openshift-default-64cdfcbbff-t9qqp\" (UID: \"62a849d6-190a-4950-953c-565117367f4d\") " pod="openshift-ingress/e2e-api-b-e60798-openshift-default-64cdfcbbff-t9qqp"
Jul 07 16:17:38.800672 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:38.800514    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62a849d6-190a-4950-953c-565117367f4d-workload-certs\") pod \"e2e-api-b-e60798-openshift-default-64cdfcbbff-t9qqp\" (UID: \"62a849d6-190a-4950-953c-565117367f4d\") " pod="openshift-ingress/e2e-api-b-e60798-openshift-default-64cdfcbbff-t9qqp"
Jul 07 16:17:38.800672 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:38.800568    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62a849d6-190a-4950-953c-565117367f4d-istio-data\") pod \"e2e-api-b-e60798-openshift-default-64cdfcbbff-t9qqp\" (UID: \"62a849d6-190a-4950-953c-565117367f4d\") " pod="openshift-ingress/e2e-api-b-e60798-openshift-default-64cdfcbbff-t9qqp"
Jul 07 16:17:38.800672 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:38.800595    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62a849d6-190a-4950-953c-565117367f4d-istiod-ca-cert\") pod \"e2e-api-b-e60798-openshift-default-64cdfcbbff-t9qqp\" (UID: \"62a849d6-190a-4950-953c-565117367f4d\") " pod="openshift-ingress/e2e-api-b-e60798-openshift-default-64cdfcbbff-t9qqp"
Jul 07 16:17:38.800672 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:38.800620    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92tsh\" (UniqueName: \"kubernetes.io/projected/62a849d6-190a-4950-953c-565117367f4d-kube-api-access-92tsh\") pod \"e2e-api-b-e60798-openshift-default-64cdfcbbff-t9qqp\" (UID: \"62a849d6-190a-4950-953c-565117367f4d\") " pod="openshift-ingress/e2e-api-b-e60798-openshift-default-64cdfcbbff-t9qqp"
Jul 07 16:17:38.800839 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:38.800730    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62a849d6-190a-4950-953c-565117367f4d-workload-socket\") pod \"e2e-api-b-e60798-openshift-default-64cdfcbbff-t9qqp\" (UID: \"62a849d6-190a-4950-953c-565117367f4d\") " pod="openshift-ingress/e2e-api-b-e60798-openshift-default-64cdfcbbff-t9qqp"
Jul 07 16:17:38.800839 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:38.800787    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62a849d6-190a-4950-953c-565117367f4d-istio-podinfo\") pod \"e2e-api-b-e60798-openshift-default-64cdfcbbff-t9qqp\" (UID: \"62a849d6-190a-4950-953c-565117367f4d\") " pod="openshift-ingress/e2e-api-b-e60798-openshift-default-64cdfcbbff-t9qqp"
Jul 07 16:17:38.800839 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:38.800820    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62a849d6-190a-4950-953c-565117367f4d-istio-token\") pod \"e2e-api-b-e60798-openshift-default-64cdfcbbff-t9qqp\" (UID: \"62a849d6-190a-4950-953c-565117367f4d\") " pod="openshift-ingress/e2e-api-b-e60798-openshift-default-64cdfcbbff-t9qqp"
Jul 07 16:17:38.800976 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:38.800843    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62a849d6-190a-4950-953c-565117367f4d-istio-envoy\") pod \"e2e-api-b-e60798-openshift-default-64cdfcbbff-t9qqp\" (UID: \"62a849d6-190a-4950-953c-565117367f4d\") " pod="openshift-ingress/e2e-api-b-e60798-openshift-default-64cdfcbbff-t9qqp"
Jul 07 16:17:38.901868 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:38.901830    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62a849d6-190a-4950-953c-565117367f4d-workload-socket\") pod \"e2e-api-b-e60798-openshift-default-64cdfcbbff-t9qqp\" (UID: \"62a849d6-190a-4950-953c-565117367f4d\") " pod="openshift-ingress/e2e-api-b-e60798-openshift-default-64cdfcbbff-t9qqp"
Jul 07 16:17:38.902084 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:38.901881    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62a849d6-190a-4950-953c-565117367f4d-istio-podinfo\") pod \"e2e-api-b-e60798-openshift-default-64cdfcbbff-t9qqp\" (UID: \"62a849d6-190a-4950-953c-565117367f4d\") " pod="openshift-ingress/e2e-api-b-e60798-openshift-default-64cdfcbbff-t9qqp"
Jul 07 16:17:38.902084 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:38.901926    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62a849d6-190a-4950-953c-565117367f4d-istio-token\") pod \"e2e-api-b-e60798-openshift-default-64cdfcbbff-t9qqp\" (UID: \"62a849d6-190a-4950-953c-565117367f4d\") " pod="openshift-ingress/e2e-api-b-e60798-openshift-default-64cdfcbbff-t9qqp"
Jul 07 16:17:38.902084 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:38.901952    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62a849d6-190a-4950-953c-565117367f4d-istio-envoy\") pod \"e2e-api-b-e60798-openshift-default-64cdfcbbff-t9qqp\" (UID: \"62a849d6-190a-4950-953c-565117367f4d\") " pod="openshift-ingress/e2e-api-b-e60798-openshift-default-64cdfcbbff-t9qqp"
Jul 07 16:17:38.902084 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:38.901993    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62a849d6-190a-4950-953c-565117367f4d-credential-socket\") pod \"e2e-api-b-e60798-openshift-default-64cdfcbbff-t9qqp\" (UID: \"62a849d6-190a-4950-953c-565117367f4d\") " pod="openshift-ingress/e2e-api-b-e60798-openshift-default-64cdfcbbff-t9qqp"
Jul 07 16:17:38.902084 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:38.902016    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62a849d6-190a-4950-953c-565117367f4d-workload-certs\") pod \"e2e-api-b-e60798-openshift-default-64cdfcbbff-t9qqp\" (UID: \"62a849d6-190a-4950-953c-565117367f4d\") " pod="openshift-ingress/e2e-api-b-e60798-openshift-default-64cdfcbbff-t9qqp"
Jul 07 16:17:38.902084 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:38.902059    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62a849d6-190a-4950-953c-565117367f4d-istio-data\") pod \"e2e-api-b-e60798-openshift-default-64cdfcbbff-t9qqp\" (UID: \"62a849d6-190a-4950-953c-565117367f4d\") " pod="openshift-ingress/e2e-api-b-e60798-openshift-default-64cdfcbbff-t9qqp"
Jul 07 16:17:38.902410 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:38.902117    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62a849d6-190a-4950-953c-565117367f4d-istiod-ca-cert\") pod \"e2e-api-b-e60798-openshift-default-64cdfcbbff-t9qqp\" (UID: \"62a849d6-190a-4950-953c-565117367f4d\") " pod="openshift-ingress/e2e-api-b-e60798-openshift-default-64cdfcbbff-t9qqp"
Jul 07 16:17:38.902410 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:38.902170    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92tsh\" (UniqueName: \"kubernetes.io/projected/62a849d6-190a-4950-953c-565117367f4d-kube-api-access-92tsh\") pod \"e2e-api-b-e60798-openshift-default-64cdfcbbff-t9qqp\" (UID: \"62a849d6-190a-4950-953c-565117367f4d\") " pod="openshift-ingress/e2e-api-b-e60798-openshift-default-64cdfcbbff-t9qqp"
Jul 07 16:17:38.902410 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:38.902252    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62a849d6-190a-4950-953c-565117367f4d-workload-socket\") pod \"e2e-api-b-e60798-openshift-default-64cdfcbbff-t9qqp\" (UID: \"62a849d6-190a-4950-953c-565117367f4d\") " pod="openshift-ingress/e2e-api-b-e60798-openshift-default-64cdfcbbff-t9qqp"
Jul 07 16:17:38.902410 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:38.902307    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62a849d6-190a-4950-953c-565117367f4d-credential-socket\") pod \"e2e-api-b-e60798-openshift-default-64cdfcbbff-t9qqp\" (UID: \"62a849d6-190a-4950-953c-565117367f4d\") " pod="openshift-ingress/e2e-api-b-e60798-openshift-default-64cdfcbbff-t9qqp"
Jul 07 16:17:38.902621 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:38.902428    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62a849d6-190a-4950-953c-565117367f4d-istio-data\") pod \"e2e-api-b-e60798-openshift-default-64cdfcbbff-t9qqp\" (UID: \"62a849d6-190a-4950-953c-565117367f4d\") " pod="openshift-ingress/e2e-api-b-e60798-openshift-default-64cdfcbbff-t9qqp"
Jul 07 16:17:38.902621 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:38.902450    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62a849d6-190a-4950-953c-565117367f4d-workload-certs\") pod \"e2e-api-b-e60798-openshift-default-64cdfcbbff-t9qqp\" (UID: \"62a849d6-190a-4950-953c-565117367f4d\") " pod="openshift-ingress/e2e-api-b-e60798-openshift-default-64cdfcbbff-t9qqp"
Jul 07 16:17:38.902827 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:38.902808    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62a849d6-190a-4950-953c-565117367f4d-istiod-ca-cert\") pod \"e2e-api-b-e60798-openshift-default-64cdfcbbff-t9qqp\" (UID: \"62a849d6-190a-4950-953c-565117367f4d\") " pod="openshift-ingress/e2e-api-b-e60798-openshift-default-64cdfcbbff-t9qqp"
Jul 07 16:17:38.904380 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:38.904355    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62a849d6-190a-4950-953c-565117367f4d-istio-envoy\") pod \"e2e-api-b-e60798-openshift-default-64cdfcbbff-t9qqp\" (UID: \"62a849d6-190a-4950-953c-565117367f4d\") " pod="openshift-ingress/e2e-api-b-e60798-openshift-default-64cdfcbbff-t9qqp"
Jul 07 16:17:38.904495 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:38.904429    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62a849d6-190a-4950-953c-565117367f4d-istio-podinfo\") pod \"e2e-api-b-e60798-openshift-default-64cdfcbbff-t9qqp\" (UID: \"62a849d6-190a-4950-953c-565117367f4d\") " pod="openshift-ingress/e2e-api-b-e60798-openshift-default-64cdfcbbff-t9qqp"
Jul 07 16:17:38.913503 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:38.913469    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92tsh\" (UniqueName: \"kubernetes.io/projected/62a849d6-190a-4950-953c-565117367f4d-kube-api-access-92tsh\") pod \"e2e-api-b-e60798-openshift-default-64cdfcbbff-t9qqp\" (UID: \"62a849d6-190a-4950-953c-565117367f4d\") " pod="openshift-ingress/e2e-api-b-e60798-openshift-default-64cdfcbbff-t9qqp"
Jul 07 16:17:38.915230 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:38.915209    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62a849d6-190a-4950-953c-565117367f4d-istio-token\") pod \"e2e-api-b-e60798-openshift-default-64cdfcbbff-t9qqp\" (UID: \"62a849d6-190a-4950-953c-565117367f4d\") " pod="openshift-ingress/e2e-api-b-e60798-openshift-default-64cdfcbbff-t9qqp"
Jul 07 16:17:39.018751 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:39.018652    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e60798-openshift-default-64cdfcbbff-t9qqp"
Jul 07 16:17:39.166578 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:39.166546    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-e60798-openshift-default-64cdfcbbff-t9qqp"]
Jul 07 16:17:39.168723 ip-10-0-138-101 kubenswrapper[2566]: W0707 16:17:39.168691    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62a849d6_190a_4950_953c_565117367f4d.slice/crio-3341cc533391dbdaa44c703b0345b61906c64be7e3eaecb2b12741fc0c65ce86 WatchSource:0}: Error finding container 3341cc533391dbdaa44c703b0345b61906c64be7e3eaecb2b12741fc0c65ce86: Status 404 returned error can't find the container with id 3341cc533391dbdaa44c703b0345b61906c64be7e3eaecb2b12741fc0c65ce86
Jul 07 16:17:39.170975 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:39.170943    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 07 16:17:39.171046 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:39.171010    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 07 16:17:39.171046 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:39.171041    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 07 16:17:39.394403 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:39.394311    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e60798-openshift-default-64cdfcbbff-t9qqp" event={"ID":"62a849d6-190a-4950-953c-565117367f4d","Type":"ContainerStarted","Data":"5d5a3b6bd9eb6db6c64cb782495aec938fc19a8537f7482a72f66458a33150ca"}
Jul 07 16:17:39.394403 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:39.394371    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e60798-openshift-default-64cdfcbbff-t9qqp" event={"ID":"62a849d6-190a-4950-953c-565117367f4d","Type":"ContainerStarted","Data":"3341cc533391dbdaa44c703b0345b61906c64be7e3eaecb2b12741fc0c65ce86"}
Jul 07 16:17:39.426587 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:39.426519    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-e60798-openshift-default-64cdfcbbff-t9qqp" podStartSLOduration=1.426497349 podStartE2EDuration="1.426497349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:17:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:17:39.425170911 +0000 UTC m=+2013.513596581" watchObservedRunningTime="2026-07-07 16:17:39.426497349 +0000 UTC m=+2013.514923019"
Jul 07 16:17:40.019680 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:40.019652    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-e60798-openshift-default-64cdfcbbff-t9qqp"
Jul 07 16:17:40.024250 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:40.024224    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-e60798-openshift-default-64cdfcbbff-t9qqp"
Jul 07 16:17:40.398640 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:40.398564    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-e60798-openshift-default-64cdfcbbff-t9qqp"
Jul 07 16:17:40.399483 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:40.399465    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-e60798-openshift-default-64cdfcbbff-t9qqp"
Jul 07 16:17:40.739944 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:40.739901    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mk99s"]
Jul 07 16:17:55.981852 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:55.980906    2566 patch_prober.go:28] interesting pod/e2e-api-b-e60798-openshift-default-64cdfcbbff-t9qqp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": read tcp 10.133.0.2:57342->10.133.0.44:15021: read: connection reset by peer" start-of-body=
Jul 07 16:17:55.981852 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:55.980992    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-e60798-openshift-default-64cdfcbbff-t9qqp" podUID="62a849d6-190a-4950-953c-565117367f4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": read tcp 10.133.0.2:57342->10.133.0.44:15021: read: connection reset by peer"
Jul 07 16:17:56.473394 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:56.473358    2566 generic.go:356] "Generic (PLEG): container finished" podID="62a849d6-190a-4950-953c-565117367f4d" containerID="5d5a3b6bd9eb6db6c64cb782495aec938fc19a8537f7482a72f66458a33150ca" exitCode=0
Jul 07 16:17:56.474797 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:56.474773    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e60798-openshift-default-64cdfcbbff-t9qqp" event={"ID":"62a849d6-190a-4950-953c-565117367f4d","Type":"ContainerDied","Data":"5d5a3b6bd9eb6db6c64cb782495aec938fc19a8537f7482a72f66458a33150ca"}
Jul 07 16:17:56.475168 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:56.475154    2566 scope.go:117] "RemoveContainer" containerID="5d5a3b6bd9eb6db6c64cb782495aec938fc19a8537f7482a72f66458a33150ca"
Jul 07 16:17:56.479030 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:56.477551    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 07 16:17:56.479030 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:56.477689    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 07 16:17:56.479030 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:56.477780    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 07 16:17:57.485334 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:57.485294    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e60798-openshift-default-64cdfcbbff-t9qqp" event={"ID":"62a849d6-190a-4950-953c-565117367f4d","Type":"ContainerStarted","Data":"d329c5c16f4120582272f8fe111ea2f1b78a0549d73623e0b79e4cfc3fd32eec"}
Jul 07 16:17:58.019768 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:58.019734    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-e60798-openshift-default-64cdfcbbff-t9qqp"
Jul 07 16:17:58.019975 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:58.019953    2566 patch_prober.go:28] interesting pod/e2e-api-b-e60798-openshift-default-64cdfcbbff-t9qqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Jul 07 16:17:58.020027 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:58.020002    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e60798-openshift-default-64cdfcbbff-t9qqp" podUID="62a849d6-190a-4950-953c-565117367f4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Jul 07 16:17:58.096435 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:58.096398    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mk99s"]
Jul 07 16:17:59.019341 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:59.019298    2566 patch_prober.go:28] interesting pod/e2e-api-b-e60798-openshift-default-64cdfcbbff-t9qqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Jul 07 16:17:59.019713 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:17:59.019364    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e60798-openshift-default-64cdfcbbff-t9qqp" podUID="62a849d6-190a-4950-953c-565117367f4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Jul 07 16:18:00.019979 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:00.019947    2566 patch_prober.go:28] interesting pod/e2e-api-b-e60798-openshift-default-64cdfcbbff-t9qqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Jul 07 16:18:00.020423 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:00.020002    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e60798-openshift-default-64cdfcbbff-t9qqp" podUID="62a849d6-190a-4950-953c-565117367f4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Jul 07 16:18:01.019691 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:01.019648    2566 patch_prober.go:28] interesting pod/e2e-api-b-e60798-openshift-default-64cdfcbbff-t9qqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Jul 07 16:18:01.019867 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:01.019708    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e60798-openshift-default-64cdfcbbff-t9qqp" podUID="62a849d6-190a-4950-953c-565117367f4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Jul 07 16:18:02.023402 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:02.023368    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-e60798-openshift-default-64cdfcbbff-t9qqp"
Jul 07 16:18:02.023812 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:02.023663    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-e60798-openshift-default-64cdfcbbff-t9qqp"
Jul 07 16:18:02.024408 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:02.024385    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-e60798-openshift-default-64cdfcbbff-t9qqp"
Jul 07 16:18:09.788954 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:09.788918    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-e60798-openshift-default-64cdfcbbff-t9qqp"]
Jul 07 16:18:09.789449 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:09.789176    2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-e60798-openshift-default-64cdfcbbff-t9qqp" podUID="62a849d6-190a-4950-953c-565117367f4d" containerName="istio-proxy" containerID="cri-o://d329c5c16f4120582272f8fe111ea2f1b78a0549d73623e0b79e4cfc3fd32eec" gracePeriod=30
Jul 07 16:18:12.836568 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:12.836533    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mk99s"]
Jul 07 16:18:15.058350 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:15.058277    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e60798-openshift-default-64cdfcbbff-t9qqp"
Jul 07 16:18:15.126266 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:15.126232    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62a849d6-190a-4950-953c-565117367f4d-istio-data\") pod \"62a849d6-190a-4950-953c-565117367f4d\" (UID: \"62a849d6-190a-4950-953c-565117367f4d\") "
Jul 07 16:18:15.126266 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:15.126270    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62a849d6-190a-4950-953c-565117367f4d-workload-socket\") pod \"62a849d6-190a-4950-953c-565117367f4d\" (UID: \"62a849d6-190a-4950-953c-565117367f4d\") "
Jul 07 16:18:15.126519 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:15.126310    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62a849d6-190a-4950-953c-565117367f4d-workload-certs\") pod \"62a849d6-190a-4950-953c-565117367f4d\" (UID: \"62a849d6-190a-4950-953c-565117367f4d\") "
Jul 07 16:18:15.126519 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:15.126327    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62a849d6-190a-4950-953c-565117367f4d-istio-podinfo\") pod \"62a849d6-190a-4950-953c-565117367f4d\" (UID: \"62a849d6-190a-4950-953c-565117367f4d\") "
Jul 07 16:18:15.126519 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:15.126350    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62a849d6-190a-4950-953c-565117367f4d-istio-token\") pod \"62a849d6-190a-4950-953c-565117367f4d\" (UID: \"62a849d6-190a-4950-953c-565117367f4d\") "
Jul 07 16:18:15.126519 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:15.126411    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62a849d6-190a-4950-953c-565117367f4d-credential-socket\") pod \"62a849d6-190a-4950-953c-565117367f4d\" (UID: \"62a849d6-190a-4950-953c-565117367f4d\") "
Jul 07 16:18:15.126519 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:15.126439    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62a849d6-190a-4950-953c-565117367f4d-istiod-ca-cert\") pod \"62a849d6-190a-4950-953c-565117367f4d\" (UID: \"62a849d6-190a-4950-953c-565117367f4d\") "
Jul 07 16:18:15.126519 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:15.126477    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-92tsh\" (UniqueName: \"kubernetes.io/projected/62a849d6-190a-4950-953c-565117367f4d-kube-api-access-92tsh\") pod \"62a849d6-190a-4950-953c-565117367f4d\" (UID: \"62a849d6-190a-4950-953c-565117367f4d\") "
Jul 07 16:18:15.126519 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:15.126498    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62a849d6-190a-4950-953c-565117367f4d-istio-envoy\") pod \"62a849d6-190a-4950-953c-565117367f4d\" (UID: \"62a849d6-190a-4950-953c-565117367f4d\") "
Jul 07 16:18:15.126876 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:15.126553    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62a849d6-190a-4950-953c-565117367f4d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "62a849d6-190a-4950-953c-565117367f4d" (UID: "62a849d6-190a-4950-953c-565117367f4d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 16:18:15.126876 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:15.126626    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62a849d6-190a-4950-953c-565117367f4d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "62a849d6-190a-4950-953c-565117367f4d" (UID: "62a849d6-190a-4950-953c-565117367f4d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 16:18:15.126876 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:15.126801    2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62a849d6-190a-4950-953c-565117367f4d-workload-socket\") on node \"ip-10-0-138-101.ec2.internal\" DevicePath \"\""
Jul 07 16:18:15.126876 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:15.126821    2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62a849d6-190a-4950-953c-565117367f4d-workload-certs\") on node \"ip-10-0-138-101.ec2.internal\" DevicePath \"\""
Jul 07 16:18:15.126876 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:15.126825    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62a849d6-190a-4950-953c-565117367f4d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "62a849d6-190a-4950-953c-565117367f4d" (UID: "62a849d6-190a-4950-953c-565117367f4d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 16:18:15.127151 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:15.127101    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62a849d6-190a-4950-953c-565117367f4d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "62a849d6-190a-4950-953c-565117367f4d" (UID: "62a849d6-190a-4950-953c-565117367f4d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 16:18:15.127294 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:15.127266    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62a849d6-190a-4950-953c-565117367f4d-istio-data" (OuterVolumeSpecName: "istio-data") pod "62a849d6-190a-4950-953c-565117367f4d" (UID: "62a849d6-190a-4950-953c-565117367f4d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 16:18:15.128622 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:15.128585    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/62a849d6-190a-4950-953c-565117367f4d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "62a849d6-190a-4950-953c-565117367f4d" (UID: "62a849d6-190a-4950-953c-565117367f4d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 07 16:18:15.128731 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:15.128664    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62a849d6-190a-4950-953c-565117367f4d-kube-api-access-92tsh" (OuterVolumeSpecName: "kube-api-access-92tsh") pod "62a849d6-190a-4950-953c-565117367f4d" (UID: "62a849d6-190a-4950-953c-565117367f4d"). InnerVolumeSpecName "kube-api-access-92tsh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 16:18:15.128731 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:15.128660    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62a849d6-190a-4950-953c-565117367f4d-istio-token" (OuterVolumeSpecName: "istio-token") pod "62a849d6-190a-4950-953c-565117367f4d" (UID: "62a849d6-190a-4950-953c-565117367f4d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 16:18:15.128731 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:15.128709    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62a849d6-190a-4950-953c-565117367f4d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "62a849d6-190a-4950-953c-565117367f4d" (UID: "62a849d6-190a-4950-953c-565117367f4d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 16:18:15.227546 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:15.227511    2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62a849d6-190a-4950-953c-565117367f4d-credential-socket\") on node \"ip-10-0-138-101.ec2.internal\" DevicePath \"\""
Jul 07 16:18:15.227546 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:15.227541    2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62a849d6-190a-4950-953c-565117367f4d-istiod-ca-cert\") on node \"ip-10-0-138-101.ec2.internal\" DevicePath \"\""
Jul 07 16:18:15.227546 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:15.227551    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92tsh\" (UniqueName: \"kubernetes.io/projected/62a849d6-190a-4950-953c-565117367f4d-kube-api-access-92tsh\") on node \"ip-10-0-138-101.ec2.internal\" DevicePath \"\""
Jul 07 16:18:15.227774 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:15.227561    2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62a849d6-190a-4950-953c-565117367f4d-istio-envoy\") on node \"ip-10-0-138-101.ec2.internal\" DevicePath \"\""
Jul 07 16:18:15.227774 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:15.227570    2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62a849d6-190a-4950-953c-565117367f4d-istio-data\") on node \"ip-10-0-138-101.ec2.internal\" DevicePath \"\""
Jul 07 16:18:15.227774 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:15.227577    2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62a849d6-190a-4950-953c-565117367f4d-istio-podinfo\") on node \"ip-10-0-138-101.ec2.internal\" DevicePath \"\""
Jul 07 16:18:15.227774 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:15.227585    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62a849d6-190a-4950-953c-565117367f4d-istio-token\") on node \"ip-10-0-138-101.ec2.internal\" DevicePath \"\""
Jul 07 16:18:15.575849 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:15.575817    2566 generic.go:356] "Generic (PLEG): container finished" podID="62a849d6-190a-4950-953c-565117367f4d" containerID="d329c5c16f4120582272f8fe111ea2f1b78a0549d73623e0b79e4cfc3fd32eec" exitCode=0
Jul 07 16:18:15.576079 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:15.575879    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e60798-openshift-default-64cdfcbbff-t9qqp"
Jul 07 16:18:15.576079 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:15.575914    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e60798-openshift-default-64cdfcbbff-t9qqp" event={"ID":"62a849d6-190a-4950-953c-565117367f4d","Type":"ContainerDied","Data":"d329c5c16f4120582272f8fe111ea2f1b78a0549d73623e0b79e4cfc3fd32eec"}
Jul 07 16:18:15.576079 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:15.575977    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e60798-openshift-default-64cdfcbbff-t9qqp" event={"ID":"62a849d6-190a-4950-953c-565117367f4d","Type":"ContainerDied","Data":"3341cc533391dbdaa44c703b0345b61906c64be7e3eaecb2b12741fc0c65ce86"}
Jul 07 16:18:15.576079 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:15.576007    2566 scope.go:117] "RemoveContainer" containerID="d329c5c16f4120582272f8fe111ea2f1b78a0549d73623e0b79e4cfc3fd32eec"
Jul 07 16:18:15.585650 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:15.585632    2566 scope.go:117] "RemoveContainer" containerID="5d5a3b6bd9eb6db6c64cb782495aec938fc19a8537f7482a72f66458a33150ca"
Jul 07 16:18:15.594326 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:15.594304    2566 scope.go:117] "RemoveContainer" containerID="d329c5c16f4120582272f8fe111ea2f1b78a0549d73623e0b79e4cfc3fd32eec"
Jul 07 16:18:15.594623 ip-10-0-138-101 kubenswrapper[2566]: E0707 16:18:15.594605    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d329c5c16f4120582272f8fe111ea2f1b78a0549d73623e0b79e4cfc3fd32eec\": container with ID starting with d329c5c16f4120582272f8fe111ea2f1b78a0549d73623e0b79e4cfc3fd32eec not found: ID does not exist" containerID="d329c5c16f4120582272f8fe111ea2f1b78a0549d73623e0b79e4cfc3fd32eec"
Jul 07 16:18:15.594673 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:15.594632    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d329c5c16f4120582272f8fe111ea2f1b78a0549d73623e0b79e4cfc3fd32eec"} err="failed to get container status \"d329c5c16f4120582272f8fe111ea2f1b78a0549d73623e0b79e4cfc3fd32eec\": rpc error: code = NotFound desc = could not find container \"d329c5c16f4120582272f8fe111ea2f1b78a0549d73623e0b79e4cfc3fd32eec\": container with ID starting with d329c5c16f4120582272f8fe111ea2f1b78a0549d73623e0b79e4cfc3fd32eec not found: ID does not exist"
Jul 07 16:18:15.594673 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:15.594650    2566 scope.go:117] "RemoveContainer" containerID="5d5a3b6bd9eb6db6c64cb782495aec938fc19a8537f7482a72f66458a33150ca"
Jul 07 16:18:15.594869 ip-10-0-138-101 kubenswrapper[2566]: E0707 16:18:15.594855    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d5a3b6bd9eb6db6c64cb782495aec938fc19a8537f7482a72f66458a33150ca\": container with ID starting with 5d5a3b6bd9eb6db6c64cb782495aec938fc19a8537f7482a72f66458a33150ca not found: ID does not exist" containerID="5d5a3b6bd9eb6db6c64cb782495aec938fc19a8537f7482a72f66458a33150ca"
Jul 07 16:18:15.594944 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:15.594873    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d5a3b6bd9eb6db6c64cb782495aec938fc19a8537f7482a72f66458a33150ca"} err="failed to get container status \"5d5a3b6bd9eb6db6c64cb782495aec938fc19a8537f7482a72f66458a33150ca\": rpc error: code = NotFound desc = could not find container \"5d5a3b6bd9eb6db6c64cb782495aec938fc19a8537f7482a72f66458a33150ca\": container with ID starting with 5d5a3b6bd9eb6db6c64cb782495aec938fc19a8537f7482a72f66458a33150ca not found: ID does not exist"
Jul 07 16:18:15.601135 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:15.601106    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-e60798-openshift-default-64cdfcbbff-t9qqp"]
Jul 07 16:18:15.610058 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:15.610032    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-e60798-openshift-default-64cdfcbbff-t9qqp"]
Jul 07 16:18:16.477177 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:16.477150    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62a849d6-190a-4950-953c-565117367f4d" path="/var/lib/kubelet/pods/62a849d6-190a-4950-953c-565117367f4d/volumes"
Jul 07 16:18:18.627952 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:18.627919    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-1cea82-openshift-default-6ff8898c79-7mshg"]
Jul 07 16:18:18.628318 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:18.628245    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62a849d6-190a-4950-953c-565117367f4d" containerName="istio-proxy"
Jul 07 16:18:18.628318 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:18.628256    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="62a849d6-190a-4950-953c-565117367f4d" containerName="istio-proxy"
Jul 07 16:18:18.628318 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:18.628271    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cebc66f0-4d97-41bd-acb3-73ddbbd88260" containerName="istio-proxy"
Jul 07 16:18:18.628318 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:18.628276    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="cebc66f0-4d97-41bd-acb3-73ddbbd88260" containerName="istio-proxy"
Jul 07 16:18:18.628459 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:18.628332    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="62a849d6-190a-4950-953c-565117367f4d" containerName="istio-proxy"
Jul 07 16:18:18.628459 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:18.628343    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="cebc66f0-4d97-41bd-acb3-73ddbbd88260" containerName="istio-proxy"
Jul 07 16:18:18.632625 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:18.632610    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-1cea82-openshift-default-6ff8898c79-7mshg"
Jul 07 16:18:18.636087 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:18.636062    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-1cea82-openshift-default-dockercfg-pqfkm\""
Jul 07 16:18:18.650635 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:18.650609    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-1cea82-openshift-default-6ff8898c79-7mshg"]
Jul 07 16:18:18.657162 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:18.657138    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f28c34bc-0591-4290-9d69-3f8a1cb95213-workload-certs\") pod \"e2e-shared-b-1cea82-openshift-default-6ff8898c79-7mshg\" (UID: \"f28c34bc-0591-4290-9d69-3f8a1cb95213\") " pod="openshift-ingress/e2e-shared-b-1cea82-openshift-default-6ff8898c79-7mshg"
Jul 07 16:18:18.657274 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:18.657189    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f28c34bc-0591-4290-9d69-3f8a1cb95213-credential-socket\") pod \"e2e-shared-b-1cea82-openshift-default-6ff8898c79-7mshg\" (UID: \"f28c34bc-0591-4290-9d69-3f8a1cb95213\") " pod="openshift-ingress/e2e-shared-b-1cea82-openshift-default-6ff8898c79-7mshg"
Jul 07 16:18:18.657274 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:18.657219    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f28c34bc-0591-4290-9d69-3f8a1cb95213-istio-data\") pod \"e2e-shared-b-1cea82-openshift-default-6ff8898c79-7mshg\" (UID: \"f28c34bc-0591-4290-9d69-3f8a1cb95213\") " pod="openshift-ingress/e2e-shared-b-1cea82-openshift-default-6ff8898c79-7mshg"
Jul 07 16:18:18.657274 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:18.657249    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f28c34bc-0591-4290-9d69-3f8a1cb95213-istiod-ca-cert\") pod \"e2e-shared-b-1cea82-openshift-default-6ff8898c79-7mshg\" (UID: \"f28c34bc-0591-4290-9d69-3f8a1cb95213\") " pod="openshift-ingress/e2e-shared-b-1cea82-openshift-default-6ff8898c79-7mshg"
Jul 07 16:18:18.657419 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:18.657292    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f28c34bc-0591-4290-9d69-3f8a1cb95213-istio-podinfo\") pod \"e2e-shared-b-1cea82-openshift-default-6ff8898c79-7mshg\" (UID: \"f28c34bc-0591-4290-9d69-3f8a1cb95213\") " pod="openshift-ingress/e2e-shared-b-1cea82-openshift-default-6ff8898c79-7mshg"
Jul 07 16:18:18.657419 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:18.657365    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcd26\" (UniqueName: \"kubernetes.io/projected/f28c34bc-0591-4290-9d69-3f8a1cb95213-kube-api-access-gcd26\") pod \"e2e-shared-b-1cea82-openshift-default-6ff8898c79-7mshg\" (UID: \"f28c34bc-0591-4290-9d69-3f8a1cb95213\") " pod="openshift-ingress/e2e-shared-b-1cea82-openshift-default-6ff8898c79-7mshg"
Jul 07 16:18:18.657419 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:18.657414    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f28c34bc-0591-4290-9d69-3f8a1cb95213-workload-socket\") pod \"e2e-shared-b-1cea82-openshift-default-6ff8898c79-7mshg\" (UID: \"f28c34bc-0591-4290-9d69-3f8a1cb95213\") " pod="openshift-ingress/e2e-shared-b-1cea82-openshift-default-6ff8898c79-7mshg"
Jul 07 16:18:18.657548 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:18.657438    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f28c34bc-0591-4290-9d69-3f8a1cb95213-istio-envoy\") pod \"e2e-shared-b-1cea82-openshift-default-6ff8898c79-7mshg\" (UID: \"f28c34bc-0591-4290-9d69-3f8a1cb95213\") " pod="openshift-ingress/e2e-shared-b-1cea82-openshift-default-6ff8898c79-7mshg"
Jul 07 16:18:18.657548 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:18.657455    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f28c34bc-0591-4290-9d69-3f8a1cb95213-istio-token\") pod \"e2e-shared-b-1cea82-openshift-default-6ff8898c79-7mshg\" (UID: \"f28c34bc-0591-4290-9d69-3f8a1cb95213\") " pod="openshift-ingress/e2e-shared-b-1cea82-openshift-default-6ff8898c79-7mshg"
Jul 07 16:18:18.757999 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:18.757960    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f28c34bc-0591-4290-9d69-3f8a1cb95213-credential-socket\") pod \"e2e-shared-b-1cea82-openshift-default-6ff8898c79-7mshg\" (UID: \"f28c34bc-0591-4290-9d69-3f8a1cb95213\") " pod="openshift-ingress/e2e-shared-b-1cea82-openshift-default-6ff8898c79-7mshg"
Jul 07 16:18:18.757999 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:18.758004    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f28c34bc-0591-4290-9d69-3f8a1cb95213-istio-data\") pod \"e2e-shared-b-1cea82-openshift-default-6ff8898c79-7mshg\" (UID: \"f28c34bc-0591-4290-9d69-3f8a1cb95213\") " pod="openshift-ingress/e2e-shared-b-1cea82-openshift-default-6ff8898c79-7mshg"
Jul 07 16:18:18.758246 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:18.758181    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f28c34bc-0591-4290-9d69-3f8a1cb95213-istiod-ca-cert\") pod \"e2e-shared-b-1cea82-openshift-default-6ff8898c79-7mshg\" (UID: \"f28c34bc-0591-4290-9d69-3f8a1cb95213\") " pod="openshift-ingress/e2e-shared-b-1cea82-openshift-default-6ff8898c79-7mshg"
Jul 07 16:18:18.758246 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:18.758243    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f28c34bc-0591-4290-9d69-3f8a1cb95213-istio-podinfo\") pod \"e2e-shared-b-1cea82-openshift-default-6ff8898c79-7mshg\" (UID: \"f28c34bc-0591-4290-9d69-3f8a1cb95213\") " pod="openshift-ingress/e2e-shared-b-1cea82-openshift-default-6ff8898c79-7mshg"
Jul 07 16:18:18.758362 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:18.758296    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcd26\" (UniqueName: \"kubernetes.io/projected/f28c34bc-0591-4290-9d69-3f8a1cb95213-kube-api-access-gcd26\") pod \"e2e-shared-b-1cea82-openshift-default-6ff8898c79-7mshg\" (UID: \"f28c34bc-0591-4290-9d69-3f8a1cb95213\") " pod="openshift-ingress/e2e-shared-b-1cea82-openshift-default-6ff8898c79-7mshg"
Jul 07 16:18:18.758362 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:18.758320    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f28c34bc-0591-4290-9d69-3f8a1cb95213-credential-socket\") pod \"e2e-shared-b-1cea82-openshift-default-6ff8898c79-7mshg\" (UID: \"f28c34bc-0591-4290-9d69-3f8a1cb95213\") " pod="openshift-ingress/e2e-shared-b-1cea82-openshift-default-6ff8898c79-7mshg"
Jul 07 16:18:18.758362 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:18.758334    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f28c34bc-0591-4290-9d69-3f8a1cb95213-workload-socket\") pod \"e2e-shared-b-1cea82-openshift-default-6ff8898c79-7mshg\" (UID: \"f28c34bc-0591-4290-9d69-3f8a1cb95213\") " pod="openshift-ingress/e2e-shared-b-1cea82-openshift-default-6ff8898c79-7mshg"
Jul 07 16:18:18.758523 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:18.758366    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f28c34bc-0591-4290-9d69-3f8a1cb95213-istio-envoy\") pod \"e2e-shared-b-1cea82-openshift-default-6ff8898c79-7mshg\" (UID: \"f28c34bc-0591-4290-9d69-3f8a1cb95213\") " pod="openshift-ingress/e2e-shared-b-1cea82-openshift-default-6ff8898c79-7mshg"
Jul 07 16:18:18.758523 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:18.758383    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f28c34bc-0591-4290-9d69-3f8a1cb95213-istio-token\") pod \"e2e-shared-b-1cea82-openshift-default-6ff8898c79-7mshg\" (UID: \"f28c34bc-0591-4290-9d69-3f8a1cb95213\") " pod="openshift-ingress/e2e-shared-b-1cea82-openshift-default-6ff8898c79-7mshg"
Jul 07 16:18:18.758523 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:18.758401    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f28c34bc-0591-4290-9d69-3f8a1cb95213-istio-data\") pod \"e2e-shared-b-1cea82-openshift-default-6ff8898c79-7mshg\" (UID: \"f28c34bc-0591-4290-9d69-3f8a1cb95213\") " pod="openshift-ingress/e2e-shared-b-1cea82-openshift-default-6ff8898c79-7mshg"
Jul 07 16:18:18.758683 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:18.758431    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f28c34bc-0591-4290-9d69-3f8a1cb95213-workload-certs\") pod \"e2e-shared-b-1cea82-openshift-default-6ff8898c79-7mshg\" (UID: \"f28c34bc-0591-4290-9d69-3f8a1cb95213\") " pod="openshift-ingress/e2e-shared-b-1cea82-openshift-default-6ff8898c79-7mshg"
Jul 07 16:18:18.758683 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:18.758650    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f28c34bc-0591-4290-9d69-3f8a1cb95213-workload-certs\") pod \"e2e-shared-b-1cea82-openshift-default-6ff8898c79-7mshg\" (UID: \"f28c34bc-0591-4290-9d69-3f8a1cb95213\") " pod="openshift-ingress/e2e-shared-b-1cea82-openshift-default-6ff8898c79-7mshg"
Jul 07 16:18:18.758683 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:18.758587    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f28c34bc-0591-4290-9d69-3f8a1cb95213-workload-socket\") pod \"e2e-shared-b-1cea82-openshift-default-6ff8898c79-7mshg\" (UID: \"f28c34bc-0591-4290-9d69-3f8a1cb95213\") " pod="openshift-ingress/e2e-shared-b-1cea82-openshift-default-6ff8898c79-7mshg"
Jul 07 16:18:18.758947 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:18.758922    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f28c34bc-0591-4290-9d69-3f8a1cb95213-istiod-ca-cert\") pod \"e2e-shared-b-1cea82-openshift-default-6ff8898c79-7mshg\" (UID: \"f28c34bc-0591-4290-9d69-3f8a1cb95213\") " pod="openshift-ingress/e2e-shared-b-1cea82-openshift-default-6ff8898c79-7mshg"
Jul 07 16:18:18.760675 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:18.760644    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f28c34bc-0591-4290-9d69-3f8a1cb95213-istio-envoy\") pod \"e2e-shared-b-1cea82-openshift-default-6ff8898c79-7mshg\" (UID: \"f28c34bc-0591-4290-9d69-3f8a1cb95213\") " pod="openshift-ingress/e2e-shared-b-1cea82-openshift-default-6ff8898c79-7mshg"
Jul 07 16:18:18.760806 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:18.760786    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f28c34bc-0591-4290-9d69-3f8a1cb95213-istio-podinfo\") pod \"e2e-shared-b-1cea82-openshift-default-6ff8898c79-7mshg\" (UID: \"f28c34bc-0591-4290-9d69-3f8a1cb95213\") " pod="openshift-ingress/e2e-shared-b-1cea82-openshift-default-6ff8898c79-7mshg"
Jul 07 16:18:18.767232 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:18.767193    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f28c34bc-0591-4290-9d69-3f8a1cb95213-istio-token\") pod \"e2e-shared-b-1cea82-openshift-default-6ff8898c79-7mshg\" (UID: \"f28c34bc-0591-4290-9d69-3f8a1cb95213\") " pod="openshift-ingress/e2e-shared-b-1cea82-openshift-default-6ff8898c79-7mshg"
Jul 07 16:18:18.767975 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:18.767908    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcd26\" (UniqueName: \"kubernetes.io/projected/f28c34bc-0591-4290-9d69-3f8a1cb95213-kube-api-access-gcd26\") pod \"e2e-shared-b-1cea82-openshift-default-6ff8898c79-7mshg\" (UID: \"f28c34bc-0591-4290-9d69-3f8a1cb95213\") " pod="openshift-ingress/e2e-shared-b-1cea82-openshift-default-6ff8898c79-7mshg"
Jul 07 16:18:18.944542 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:18.944454    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-1cea82-openshift-default-6ff8898c79-7mshg"
Jul 07 16:18:19.085795 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:19.085747    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-1cea82-openshift-default-6ff8898c79-7mshg"]
Jul 07 16:18:19.089235 ip-10-0-138-101 kubenswrapper[2566]: W0707 16:18:19.089195    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf28c34bc_0591_4290_9d69_3f8a1cb95213.slice/crio-22a4b2a8545d0d4d98d994e0e2f23dff98fb15377e567faa0814791bda1275e3 WatchSource:0}: Error finding container 22a4b2a8545d0d4d98d994e0e2f23dff98fb15377e567faa0814791bda1275e3: Status 404 returned error can't find the container with id 22a4b2a8545d0d4d98d994e0e2f23dff98fb15377e567faa0814791bda1275e3
Jul 07 16:18:19.091913 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:19.091852    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 07 16:18:19.092000 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:19.091962    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 07 16:18:19.092049 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:19.092004    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 07 16:18:19.596616 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:19.596578    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-1cea82-openshift-default-6ff8898c79-7mshg" event={"ID":"f28c34bc-0591-4290-9d69-3f8a1cb95213","Type":"ContainerStarted","Data":"e4270e88f42b2a38a291730bb377dc4a3e998cc54e8d09eb53a9f824295abdd8"}
Jul 07 16:18:19.596814 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:19.596628    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-1cea82-openshift-default-6ff8898c79-7mshg" event={"ID":"f28c34bc-0591-4290-9d69-3f8a1cb95213","Type":"ContainerStarted","Data":"22a4b2a8545d0d4d98d994e0e2f23dff98fb15377e567faa0814791bda1275e3"}
Jul 07 16:18:19.642079 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:19.642023    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-1cea82-openshift-default-6ff8898c79-7mshg" podStartSLOduration=1.642007681 podStartE2EDuration="1.642007681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:18:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:18:19.639173457 +0000 UTC m=+2053.727599128" watchObservedRunningTime="2026-07-07 16:18:19.642007681 +0000 UTC m=+2053.730433342"
Jul 07 16:18:19.945638 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:19.945547    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-1cea82-openshift-default-6ff8898c79-7mshg"
Jul 07 16:18:20.950772 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:20.950741    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-1cea82-openshift-default-6ff8898c79-7mshg"
Jul 07 16:18:21.605987 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:21.605949    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-1cea82-openshift-default-6ff8898c79-7mshg"
Jul 07 16:18:21.606995 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:21.606970    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-1cea82-openshift-default-6ff8898c79-7mshg"
Jul 07 16:18:23.635127 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:23.635098    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mk99s"]
Jul 07 16:18:33.528620 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:33.528583    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mk99s"]
Jul 07 16:18:35.675946 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:35.675838    2566 generic.go:356] "Generic (PLEG): container finished" podID="f28c34bc-0591-4290-9d69-3f8a1cb95213" containerID="e4270e88f42b2a38a291730bb377dc4a3e998cc54e8d09eb53a9f824295abdd8" exitCode=0
Jul 07 16:18:35.675946 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:35.675912    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-1cea82-openshift-default-6ff8898c79-7mshg" event={"ID":"f28c34bc-0591-4290-9d69-3f8a1cb95213","Type":"ContainerDied","Data":"e4270e88f42b2a38a291730bb377dc4a3e998cc54e8d09eb53a9f824295abdd8"}
Jul 07 16:18:35.676420 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:35.676308    2566 scope.go:117] "RemoveContainer" containerID="e4270e88f42b2a38a291730bb377dc4a3e998cc54e8d09eb53a9f824295abdd8"
Jul 07 16:18:35.677314 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:35.677283    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 07 16:18:35.677392 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:35.677358    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 07 16:18:35.677392 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:35.677388    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 07 16:18:36.606487 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:36.606452    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-1cea82-openshift-default-6ff8898c79-7mshg"
Jul 07 16:18:36.684854 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:36.684809    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-1cea82-openshift-default-6ff8898c79-7mshg" event={"ID":"f28c34bc-0591-4290-9d69-3f8a1cb95213","Type":"ContainerStarted","Data":"65c12247db59c5aee938b9789d4d297d13d380fa11448e60593099ff1e5e7680"}
Jul 07 16:18:36.944693 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:36.944610    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-1cea82-openshift-default-6ff8898c79-7mshg"
Jul 07 16:18:36.945735 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:36.945707    2566 patch_prober.go:28] interesting pod/e2e-shared-b-1cea82-openshift-default-6ff8898c79-7mshg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Jul 07 16:18:36.945863 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:36.945759    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-1cea82-openshift-default-6ff8898c79-7mshg" podUID="f28c34bc-0591-4290-9d69-3f8a1cb95213" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Jul 07 16:18:37.945263 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:37.945227    2566 patch_prober.go:28] interesting pod/e2e-shared-b-1cea82-openshift-default-6ff8898c79-7mshg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Jul 07 16:18:37.945653 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:37.945284    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-1cea82-openshift-default-6ff8898c79-7mshg" podUID="f28c34bc-0591-4290-9d69-3f8a1cb95213" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Jul 07 16:18:38.945556 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:38.945519    2566 patch_prober.go:28] interesting pod/e2e-shared-b-1cea82-openshift-default-6ff8898c79-7mshg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Jul 07 16:18:38.945964 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:38.945574    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-1cea82-openshift-default-6ff8898c79-7mshg" podUID="f28c34bc-0591-4290-9d69-3f8a1cb95213" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Jul 07 16:18:39.945518 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:39.945481    2566 patch_prober.go:28] interesting pod/e2e-shared-b-1cea82-openshift-default-6ff8898c79-7mshg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Jul 07 16:18:39.945695 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:39.945539    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-1cea82-openshift-default-6ff8898c79-7mshg" podUID="f28c34bc-0591-4290-9d69-3f8a1cb95213" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Jul 07 16:18:40.949143 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:40.949112    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-1cea82-openshift-default-6ff8898c79-7mshg"
Jul 07 16:18:40.949573 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:40.949357    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-1cea82-openshift-default-6ff8898c79-7mshg"
Jul 07 16:18:40.950024 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:40.950007    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-1cea82-openshift-default-6ff8898c79-7mshg"
Jul 07 16:18:41.634049 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:41.634014    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-94a996/auth-test-model-94a996-kserve-84649cc75-r5whk"]
Jul 07 16:18:41.634465 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:41.634447    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62a849d6-190a-4950-953c-565117367f4d" containerName="istio-proxy"
Jul 07 16:18:41.634555 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:41.634467    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="62a849d6-190a-4950-953c-565117367f4d" containerName="istio-proxy"
Jul 07 16:18:41.634609 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:41.634567    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="62a849d6-190a-4950-953c-565117367f4d" containerName="istio-proxy"
Jul 07 16:18:41.637535 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:41.637514    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-94a996/auth-test-model-94a996-kserve-84649cc75-r5whk"
Jul 07 16:18:41.640873 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:41.640852    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-94a996\"/\"auth-test-model-94a996-kserve-self-signed-certs\""
Jul 07 16:18:41.640873 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:41.640860    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-94a996\"/\"default-dockercfg-2tt2z\""
Jul 07 16:18:41.641080 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:41.640918    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-94a996\"/\"kube-root-ca.crt\""
Jul 07 16:18:41.641080 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:41.640972    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-94a996\"/\"openshift-service-ca.crt\""
Jul 07 16:18:41.648154 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:41.648131    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-94a996/auth-test-model-94a996-kserve-84649cc75-r5whk"]
Jul 07 16:18:41.770316 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:41.770274    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87aeae47-0875-4102-8700-ae4b899cabbc-home\") pod \"auth-test-model-94a996-kserve-84649cc75-r5whk\" (UID: \"87aeae47-0875-4102-8700-ae4b899cabbc\") " pod="ai-tenant-e2e-shared-a-94a996/auth-test-model-94a996-kserve-84649cc75-r5whk"
Jul 07 16:18:41.770479 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:41.770445    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87aeae47-0875-4102-8700-ae4b899cabbc-model-cache\") pod \"auth-test-model-94a996-kserve-84649cc75-r5whk\" (UID: \"87aeae47-0875-4102-8700-ae4b899cabbc\") " pod="ai-tenant-e2e-shared-a-94a996/auth-test-model-94a996-kserve-84649cc75-r5whk"
Jul 07 16:18:41.770568 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:41.770548    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvb7m\" (UniqueName: \"kubernetes.io/projected/87aeae47-0875-4102-8700-ae4b899cabbc-kube-api-access-fvb7m\") pod \"auth-test-model-94a996-kserve-84649cc75-r5whk\" (UID: \"87aeae47-0875-4102-8700-ae4b899cabbc\") " pod="ai-tenant-e2e-shared-a-94a996/auth-test-model-94a996-kserve-84649cc75-r5whk"
Jul 07 16:18:41.770648 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:41.770627    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87aeae47-0875-4102-8700-ae4b899cabbc-tmp-dir\") pod \"auth-test-model-94a996-kserve-84649cc75-r5whk\" (UID: \"87aeae47-0875-4102-8700-ae4b899cabbc\") " pod="ai-tenant-e2e-shared-a-94a996/auth-test-model-94a996-kserve-84649cc75-r5whk"
Jul 07 16:18:41.770728 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:41.770711    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87aeae47-0875-4102-8700-ae4b899cabbc-dshm\") pod \"auth-test-model-94a996-kserve-84649cc75-r5whk\" (UID: \"87aeae47-0875-4102-8700-ae4b899cabbc\") " pod="ai-tenant-e2e-shared-a-94a996/auth-test-model-94a996-kserve-84649cc75-r5whk"
Jul 07 16:18:41.770787 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:41.770743    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87aeae47-0875-4102-8700-ae4b899cabbc-tls-certs\") pod \"auth-test-model-94a996-kserve-84649cc75-r5whk\" (UID: \"87aeae47-0875-4102-8700-ae4b899cabbc\") " pod="ai-tenant-e2e-shared-a-94a996/auth-test-model-94a996-kserve-84649cc75-r5whk"
Jul 07 16:18:41.871765 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:41.871728    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvb7m\" (UniqueName: \"kubernetes.io/projected/87aeae47-0875-4102-8700-ae4b899cabbc-kube-api-access-fvb7m\") pod \"auth-test-model-94a996-kserve-84649cc75-r5whk\" (UID: \"87aeae47-0875-4102-8700-ae4b899cabbc\") " pod="ai-tenant-e2e-shared-a-94a996/auth-test-model-94a996-kserve-84649cc75-r5whk"
Jul 07 16:18:41.871765 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:41.871767    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87aeae47-0875-4102-8700-ae4b899cabbc-tmp-dir\") pod \"auth-test-model-94a996-kserve-84649cc75-r5whk\" (UID: \"87aeae47-0875-4102-8700-ae4b899cabbc\") " pod="ai-tenant-e2e-shared-a-94a996/auth-test-model-94a996-kserve-84649cc75-r5whk"
Jul 07 16:18:41.871989 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:41.871927    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87aeae47-0875-4102-8700-ae4b899cabbc-dshm\") pod \"auth-test-model-94a996-kserve-84649cc75-r5whk\" (UID: \"87aeae47-0875-4102-8700-ae4b899cabbc\") " pod="ai-tenant-e2e-shared-a-94a996/auth-test-model-94a996-kserve-84649cc75-r5whk"
Jul 07 16:18:41.871989 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:41.871960    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87aeae47-0875-4102-8700-ae4b899cabbc-tls-certs\") pod \"auth-test-model-94a996-kserve-84649cc75-r5whk\" (UID: \"87aeae47-0875-4102-8700-ae4b899cabbc\") " pod="ai-tenant-e2e-shared-a-94a996/auth-test-model-94a996-kserve-84649cc75-r5whk"
Jul 07 16:18:41.871989 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:41.871984    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87aeae47-0875-4102-8700-ae4b899cabbc-home\") pod \"auth-test-model-94a996-kserve-84649cc75-r5whk\" (UID: \"87aeae47-0875-4102-8700-ae4b899cabbc\") " pod="ai-tenant-e2e-shared-a-94a996/auth-test-model-94a996-kserve-84649cc75-r5whk"
Jul 07 16:18:41.872143 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:41.872044    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87aeae47-0875-4102-8700-ae4b899cabbc-model-cache\") pod \"auth-test-model-94a996-kserve-84649cc75-r5whk\" (UID: \"87aeae47-0875-4102-8700-ae4b899cabbc\") " pod="ai-tenant-e2e-shared-a-94a996/auth-test-model-94a996-kserve-84649cc75-r5whk"
Jul 07 16:18:41.872382 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:41.872360    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87aeae47-0875-4102-8700-ae4b899cabbc-tmp-dir\") pod \"auth-test-model-94a996-kserve-84649cc75-r5whk\" (UID: \"87aeae47-0875-4102-8700-ae4b899cabbc\") " pod="ai-tenant-e2e-shared-a-94a996/auth-test-model-94a996-kserve-84649cc75-r5whk"
Jul 07 16:18:41.872483 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:41.872379    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87aeae47-0875-4102-8700-ae4b899cabbc-home\") pod \"auth-test-model-94a996-kserve-84649cc75-r5whk\" (UID: \"87aeae47-0875-4102-8700-ae4b899cabbc\") " pod="ai-tenant-e2e-shared-a-94a996/auth-test-model-94a996-kserve-84649cc75-r5whk"
Jul 07 16:18:41.872483 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:41.872440    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87aeae47-0875-4102-8700-ae4b899cabbc-model-cache\") pod \"auth-test-model-94a996-kserve-84649cc75-r5whk\" (UID: \"87aeae47-0875-4102-8700-ae4b899cabbc\") " pod="ai-tenant-e2e-shared-a-94a996/auth-test-model-94a996-kserve-84649cc75-r5whk"
Jul 07 16:18:41.874266 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:41.874242    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87aeae47-0875-4102-8700-ae4b899cabbc-dshm\") pod \"auth-test-model-94a996-kserve-84649cc75-r5whk\" (UID: \"87aeae47-0875-4102-8700-ae4b899cabbc\") " pod="ai-tenant-e2e-shared-a-94a996/auth-test-model-94a996-kserve-84649cc75-r5whk"
Jul 07 16:18:41.874810 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:41.874788    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87aeae47-0875-4102-8700-ae4b899cabbc-tls-certs\") pod \"auth-test-model-94a996-kserve-84649cc75-r5whk\" (UID: \"87aeae47-0875-4102-8700-ae4b899cabbc\") " pod="ai-tenant-e2e-shared-a-94a996/auth-test-model-94a996-kserve-84649cc75-r5whk"
Jul 07 16:18:41.881265 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:41.881239    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvb7m\" (UniqueName: \"kubernetes.io/projected/87aeae47-0875-4102-8700-ae4b899cabbc-kube-api-access-fvb7m\") pod \"auth-test-model-94a996-kserve-84649cc75-r5whk\" (UID: \"87aeae47-0875-4102-8700-ae4b899cabbc\") " pod="ai-tenant-e2e-shared-a-94a996/auth-test-model-94a996-kserve-84649cc75-r5whk"
Jul 07 16:18:41.947322 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:41.947223    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-94a996/auth-test-model-94a996-kserve-84649cc75-r5whk"
Jul 07 16:18:42.083809 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:42.083776    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-94a996/auth-test-model-94a996-kserve-84649cc75-r5whk"]
Jul 07 16:18:42.086811 ip-10-0-138-101 kubenswrapper[2566]: W0707 16:18:42.086785    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod87aeae47_0875_4102_8700_ae4b899cabbc.slice/crio-ebbf31c6a8b8c872b89686cfd7d4a81f1b31c4f801125dade2413e8eff4bd428 WatchSource:0}: Error finding container ebbf31c6a8b8c872b89686cfd7d4a81f1b31c4f801125dade2413e8eff4bd428: Status 404 returned error can't find the container with id ebbf31c6a8b8c872b89686cfd7d4a81f1b31c4f801125dade2413e8eff4bd428
Jul 07 16:18:42.718620 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:42.718583    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-94a996/auth-test-model-94a996-kserve-84649cc75-r5whk" event={"ID":"87aeae47-0875-4102-8700-ae4b899cabbc","Type":"ContainerStarted","Data":"dffabf4d2faba1a6b88ac8d45095810e9a572e92697a121526269ffe97e83837"}
Jul 07 16:18:42.718620 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:42.718629    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-94a996/auth-test-model-94a996-kserve-84649cc75-r5whk" event={"ID":"87aeae47-0875-4102-8700-ae4b899cabbc","Type":"ContainerStarted","Data":"ebbf31c6a8b8c872b89686cfd7d4a81f1b31c4f801125dade2413e8eff4bd428"}
Jul 07 16:18:42.740087 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:42.740010    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-94a996/auth-test-model-94a996-kserve-84649cc75-r5whk" podStartSLOduration=1.739995112 podStartE2EDuration="1.739995112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:18:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:18:42.739297841 +0000 UTC m=+2076.827723511" watchObservedRunningTime="2026-07-07 16:18:42.739995112 +0000 UTC m=+2076.828420781"
Jul 07 16:18:45.227401 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:45.227367    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mk99s"]
Jul 07 16:18:51.947953 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:51.947917    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-94a996/auth-test-model-94a996-kserve-84649cc75-r5whk"
Jul 07 16:18:51.948336 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:51.947970    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-94a996/auth-test-model-94a996-kserve-84649cc75-r5whk"
Jul 07 16:18:51.960340 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:51.960310    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-94a996/auth-test-model-94a996-kserve-84649cc75-r5whk"
Jul 07 16:18:52.775523 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:18:52.775495    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-94a996/auth-test-model-94a996-kserve-84649cc75-r5whk"
Jul 07 16:19:19.439804 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:19:19.439772    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-1cea82/auth-test-model-1cea82-kserve-5d754d768c-j7vnv"]
Jul 07 16:19:19.443220 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:19:19.443201    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1cea82/auth-test-model-1cea82-kserve-5d754d768c-j7vnv"
Jul 07 16:19:19.446500 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:19:19.446478    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-1cea82\"/\"default-dockercfg-q6lbs\""
Jul 07 16:19:19.446500 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:19:19.446486    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-1cea82\"/\"auth-test-model-1cea82-kserve-self-signed-certs\""
Jul 07 16:19:19.446675 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:19:19.446522    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-1cea82\"/\"kube-root-ca.crt\""
Jul 07 16:19:19.446675 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:19:19.446609    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-1cea82\"/\"openshift-service-ca.crt\""
Jul 07 16:19:19.455130 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:19:19.455111    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-1cea82/auth-test-model-1cea82-kserve-5d754d768c-j7vnv"]
Jul 07 16:19:19.509690 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:19:19.509654    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/670a7180-ffe0-439a-ac0f-69a9db6b9b54-home\") pod \"auth-test-model-1cea82-kserve-5d754d768c-j7vnv\" (UID: \"670a7180-ffe0-439a-ac0f-69a9db6b9b54\") " pod="ai-tenant-e2e-shared-b-1cea82/auth-test-model-1cea82-kserve-5d754d768c-j7vnv"
Jul 07 16:19:19.509857 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:19:19.509714    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7hwt\" (UniqueName: \"kubernetes.io/projected/670a7180-ffe0-439a-ac0f-69a9db6b9b54-kube-api-access-f7hwt\") pod \"auth-test-model-1cea82-kserve-5d754d768c-j7vnv\" (UID: \"670a7180-ffe0-439a-ac0f-69a9db6b9b54\") " pod="ai-tenant-e2e-shared-b-1cea82/auth-test-model-1cea82-kserve-5d754d768c-j7vnv"
Jul 07 16:19:19.509857 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:19:19.509797    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/670a7180-ffe0-439a-ac0f-69a9db6b9b54-dshm\") pod \"auth-test-model-1cea82-kserve-5d754d768c-j7vnv\" (UID: \"670a7180-ffe0-439a-ac0f-69a9db6b9b54\") " pod="ai-tenant-e2e-shared-b-1cea82/auth-test-model-1cea82-kserve-5d754d768c-j7vnv"
Jul 07 16:19:19.509993 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:19:19.509858    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/670a7180-ffe0-439a-ac0f-69a9db6b9b54-tmp-dir\") pod \"auth-test-model-1cea82-kserve-5d754d768c-j7vnv\" (UID: \"670a7180-ffe0-439a-ac0f-69a9db6b9b54\") " pod="ai-tenant-e2e-shared-b-1cea82/auth-test-model-1cea82-kserve-5d754d768c-j7vnv"
Jul 07 16:19:19.510050 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:19:19.509989    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/670a7180-ffe0-439a-ac0f-69a9db6b9b54-model-cache\") pod \"auth-test-model-1cea82-kserve-5d754d768c-j7vnv\" (UID: \"670a7180-ffe0-439a-ac0f-69a9db6b9b54\") " pod="ai-tenant-e2e-shared-b-1cea82/auth-test-model-1cea82-kserve-5d754d768c-j7vnv"
Jul 07 16:19:19.510050 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:19:19.510018    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/670a7180-ffe0-439a-ac0f-69a9db6b9b54-tls-certs\") pod \"auth-test-model-1cea82-kserve-5d754d768c-j7vnv\" (UID: \"670a7180-ffe0-439a-ac0f-69a9db6b9b54\") " pod="ai-tenant-e2e-shared-b-1cea82/auth-test-model-1cea82-kserve-5d754d768c-j7vnv"
Jul 07 16:19:19.611328 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:19:19.611271    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7hwt\" (UniqueName: \"kubernetes.io/projected/670a7180-ffe0-439a-ac0f-69a9db6b9b54-kube-api-access-f7hwt\") pod \"auth-test-model-1cea82-kserve-5d754d768c-j7vnv\" (UID: \"670a7180-ffe0-439a-ac0f-69a9db6b9b54\") " pod="ai-tenant-e2e-shared-b-1cea82/auth-test-model-1cea82-kserve-5d754d768c-j7vnv"
Jul 07 16:19:19.611563 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:19:19.611346    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/670a7180-ffe0-439a-ac0f-69a9db6b9b54-dshm\") pod \"auth-test-model-1cea82-kserve-5d754d768c-j7vnv\" (UID: \"670a7180-ffe0-439a-ac0f-69a9db6b9b54\") " pod="ai-tenant-e2e-shared-b-1cea82/auth-test-model-1cea82-kserve-5d754d768c-j7vnv"
Jul 07 16:19:19.611563 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:19:19.611414    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/670a7180-ffe0-439a-ac0f-69a9db6b9b54-tmp-dir\") pod \"auth-test-model-1cea82-kserve-5d754d768c-j7vnv\" (UID: \"670a7180-ffe0-439a-ac0f-69a9db6b9b54\") " pod="ai-tenant-e2e-shared-b-1cea82/auth-test-model-1cea82-kserve-5d754d768c-j7vnv"
Jul 07 16:19:19.611563 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:19:19.611453    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/670a7180-ffe0-439a-ac0f-69a9db6b9b54-model-cache\") pod \"auth-test-model-1cea82-kserve-5d754d768c-j7vnv\" (UID: \"670a7180-ffe0-439a-ac0f-69a9db6b9b54\") " pod="ai-tenant-e2e-shared-b-1cea82/auth-test-model-1cea82-kserve-5d754d768c-j7vnv"
Jul 07 16:19:19.611563 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:19:19.611474    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/670a7180-ffe0-439a-ac0f-69a9db6b9b54-tls-certs\") pod \"auth-test-model-1cea82-kserve-5d754d768c-j7vnv\" (UID: \"670a7180-ffe0-439a-ac0f-69a9db6b9b54\") " pod="ai-tenant-e2e-shared-b-1cea82/auth-test-model-1cea82-kserve-5d754d768c-j7vnv"
Jul 07 16:19:19.611563 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:19:19.611547    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/670a7180-ffe0-439a-ac0f-69a9db6b9b54-home\") pod \"auth-test-model-1cea82-kserve-5d754d768c-j7vnv\" (UID: \"670a7180-ffe0-439a-ac0f-69a9db6b9b54\") " pod="ai-tenant-e2e-shared-b-1cea82/auth-test-model-1cea82-kserve-5d754d768c-j7vnv"
Jul 07 16:19:19.611978 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:19:19.611949    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/670a7180-ffe0-439a-ac0f-69a9db6b9b54-tmp-dir\") pod \"auth-test-model-1cea82-kserve-5d754d768c-j7vnv\" (UID: \"670a7180-ffe0-439a-ac0f-69a9db6b9b54\") " pod="ai-tenant-e2e-shared-b-1cea82/auth-test-model-1cea82-kserve-5d754d768c-j7vnv"
Jul 07 16:19:19.611978 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:19:19.611971    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/670a7180-ffe0-439a-ac0f-69a9db6b9b54-model-cache\") pod \"auth-test-model-1cea82-kserve-5d754d768c-j7vnv\" (UID: \"670a7180-ffe0-439a-ac0f-69a9db6b9b54\") " pod="ai-tenant-e2e-shared-b-1cea82/auth-test-model-1cea82-kserve-5d754d768c-j7vnv"
Jul 07 16:19:19.612106 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:19:19.611999    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/670a7180-ffe0-439a-ac0f-69a9db6b9b54-home\") pod \"auth-test-model-1cea82-kserve-5d754d768c-j7vnv\" (UID: \"670a7180-ffe0-439a-ac0f-69a9db6b9b54\") " pod="ai-tenant-e2e-shared-b-1cea82/auth-test-model-1cea82-kserve-5d754d768c-j7vnv"
Jul 07 16:19:19.613611 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:19:19.613588    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/670a7180-ffe0-439a-ac0f-69a9db6b9b54-dshm\") pod \"auth-test-model-1cea82-kserve-5d754d768c-j7vnv\" (UID: \"670a7180-ffe0-439a-ac0f-69a9db6b9b54\") " pod="ai-tenant-e2e-shared-b-1cea82/auth-test-model-1cea82-kserve-5d754d768c-j7vnv"
Jul 07 16:19:19.613897 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:19:19.613866    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/670a7180-ffe0-439a-ac0f-69a9db6b9b54-tls-certs\") pod \"auth-test-model-1cea82-kserve-5d754d768c-j7vnv\" (UID: \"670a7180-ffe0-439a-ac0f-69a9db6b9b54\") " pod="ai-tenant-e2e-shared-b-1cea82/auth-test-model-1cea82-kserve-5d754d768c-j7vnv"
Jul 07 16:19:19.620584 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:19:19.620558    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7hwt\" (UniqueName: \"kubernetes.io/projected/670a7180-ffe0-439a-ac0f-69a9db6b9b54-kube-api-access-f7hwt\") pod \"auth-test-model-1cea82-kserve-5d754d768c-j7vnv\" (UID: \"670a7180-ffe0-439a-ac0f-69a9db6b9b54\") " pod="ai-tenant-e2e-shared-b-1cea82/auth-test-model-1cea82-kserve-5d754d768c-j7vnv"
Jul 07 16:19:19.751822 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:19:19.751783    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1cea82/auth-test-model-1cea82-kserve-5d754d768c-j7vnv"
Jul 07 16:19:19.892827 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:19:19.892801    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-1cea82/auth-test-model-1cea82-kserve-5d754d768c-j7vnv"]
Jul 07 16:19:19.894600 ip-10-0-138-101 kubenswrapper[2566]: W0707 16:19:19.894563    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod670a7180_ffe0_439a_ac0f_69a9db6b9b54.slice/crio-982763aa47dab58c468502982fdc7f6a2dfcca70e07d95cdb68e300959d905ae WatchSource:0}: Error finding container 982763aa47dab58c468502982fdc7f6a2dfcca70e07d95cdb68e300959d905ae: Status 404 returned error can't find the container with id 982763aa47dab58c468502982fdc7f6a2dfcca70e07d95cdb68e300959d905ae
Jul 07 16:19:20.889312 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:19:20.889274    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1cea82/auth-test-model-1cea82-kserve-5d754d768c-j7vnv" event={"ID":"670a7180-ffe0-439a-ac0f-69a9db6b9b54","Type":"ContainerStarted","Data":"189722d1d275e964c3100444e9fbb4d3204fc18e5693e23a0de291ce2ba11e8b"}
Jul 07 16:19:20.889683 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:19:20.889321    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1cea82/auth-test-model-1cea82-kserve-5d754d768c-j7vnv" event={"ID":"670a7180-ffe0-439a-ac0f-69a9db6b9b54","Type":"ContainerStarted","Data":"982763aa47dab58c468502982fdc7f6a2dfcca70e07d95cdb68e300959d905ae"}
Jul 07 16:19:20.908104 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:19:20.908051    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-1cea82/auth-test-model-1cea82-kserve-5d754d768c-j7vnv" podStartSLOduration=1.9080341 podStartE2EDuration="1.9080341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:19:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:19:20.906986462 +0000 UTC m=+2114.995412132" watchObservedRunningTime="2026-07-07 16:19:20.9080341 +0000 UTC m=+2114.996459782"
Jul 07 16:19:22.645230 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:19:22.645192    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mk99s"]
Jul 07 16:19:29.752767 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:19:29.752728    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-1cea82/auth-test-model-1cea82-kserve-5d754d768c-j7vnv"
Jul 07 16:19:29.753217 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:19:29.752789    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-1cea82/auth-test-model-1cea82-kserve-5d754d768c-j7vnv"
Jul 07 16:19:29.765233 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:19:29.765206    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-1cea82/auth-test-model-1cea82-kserve-5d754d768c-j7vnv"
Jul 07 16:19:29.940575 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:19:29.940543    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-1cea82/auth-test-model-1cea82-kserve-5d754d768c-j7vnv"
Jul 07 16:19:47.838961 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:19:47.838920    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mk99s"]
Jul 07 16:20:01.243202 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:20:01.243120    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mk99s"]
Jul 07 16:20:18.327112 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:20:18.327076    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mk99s"]
Jul 07 16:20:36.534822 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:20:36.534782    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mk99s"]
Jul 07 16:20:51.839371 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:20:51.839331    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mk99s"]
Jul 07 16:21:09.741721 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:21:09.741677    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mk99s"]
Jul 07 16:21:25.528143 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:21:25.528102    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mk99s"]
Jul 07 16:21:29.415857 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:21:29.415757    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-94a996/auth-test-model-94a996-kserve-84649cc75-r5whk"]
Jul 07 16:21:30.521710 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:21:30.521677    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-1cea82/auth-test-model-1cea82-kserve-5d754d768c-j7vnv"]
Jul 07 16:21:41.934349 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:21:41.934310    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mk99s"]
Jul 07 16:21:44.436206 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:21:44.436162    2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-94a996/auth-test-model-94a996-kserve-84649cc75-r5whk" podUID="87aeae47-0875-4102-8700-ae4b899cabbc" containerName="main" containerID="cri-o://dffabf4d2faba1a6b88ac8d45095810e9a572e92697a121526269ffe97e83837" gracePeriod=45
Jul 07 16:21:44.578650 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:21:44.578615    2566 generic.go:356] "Generic (PLEG): container finished" podID="87aeae47-0875-4102-8700-ae4b899cabbc" containerID="dffabf4d2faba1a6b88ac8d45095810e9a572e92697a121526269ffe97e83837" exitCode=0
Jul 07 16:21:44.578849 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:21:44.578699    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-94a996/auth-test-model-94a996-kserve-84649cc75-r5whk" event={"ID":"87aeae47-0875-4102-8700-ae4b899cabbc","Type":"ContainerDied","Data":"dffabf4d2faba1a6b88ac8d45095810e9a572e92697a121526269ffe97e83837"}
Jul 07 16:21:44.699199 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:21:44.699133    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-94a996/auth-test-model-94a996-kserve-84649cc75-r5whk"
Jul 07 16:21:44.804682 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:21:44.804643    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87aeae47-0875-4102-8700-ae4b899cabbc-model-cache\") pod \"87aeae47-0875-4102-8700-ae4b899cabbc\" (UID: \"87aeae47-0875-4102-8700-ae4b899cabbc\") "
Jul 07 16:21:44.804682 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:21:44.804688    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87aeae47-0875-4102-8700-ae4b899cabbc-tmp-dir\") pod \"87aeae47-0875-4102-8700-ae4b899cabbc\" (UID: \"87aeae47-0875-4102-8700-ae4b899cabbc\") "
Jul 07 16:21:44.805001 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:21:44.804745    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fvb7m\" (UniqueName: \"kubernetes.io/projected/87aeae47-0875-4102-8700-ae4b899cabbc-kube-api-access-fvb7m\") pod \"87aeae47-0875-4102-8700-ae4b899cabbc\" (UID: \"87aeae47-0875-4102-8700-ae4b899cabbc\") "
Jul 07 16:21:44.805001 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:21:44.804818    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87aeae47-0875-4102-8700-ae4b899cabbc-dshm\") pod \"87aeae47-0875-4102-8700-ae4b899cabbc\" (UID: \"87aeae47-0875-4102-8700-ae4b899cabbc\") "
Jul 07 16:21:44.805001 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:21:44.804854    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87aeae47-0875-4102-8700-ae4b899cabbc-tls-certs\") pod \"87aeae47-0875-4102-8700-ae4b899cabbc\" (UID: \"87aeae47-0875-4102-8700-ae4b899cabbc\") "
Jul 07 16:21:44.805001 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:21:44.804880    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87aeae47-0875-4102-8700-ae4b899cabbc-home\") pod \"87aeae47-0875-4102-8700-ae4b899cabbc\" (UID: \"87aeae47-0875-4102-8700-ae4b899cabbc\") "
Jul 07 16:21:44.805001 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:21:44.804966    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87aeae47-0875-4102-8700-ae4b899cabbc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "87aeae47-0875-4102-8700-ae4b899cabbc" (UID: "87aeae47-0875-4102-8700-ae4b899cabbc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 16:21:44.805001 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:21:44.804991    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87aeae47-0875-4102-8700-ae4b899cabbc-model-cache" (OuterVolumeSpecName: "model-cache") pod "87aeae47-0875-4102-8700-ae4b899cabbc" (UID: "87aeae47-0875-4102-8700-ae4b899cabbc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 16:21:44.805285 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:21:44.805205    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87aeae47-0875-4102-8700-ae4b899cabbc-model-cache\") on node \"ip-10-0-138-101.ec2.internal\" DevicePath \"\""
Jul 07 16:21:44.805285 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:21:44.805224    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87aeae47-0875-4102-8700-ae4b899cabbc-tmp-dir\") on node \"ip-10-0-138-101.ec2.internal\" DevicePath \"\""
Jul 07 16:21:44.805285 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:21:44.805231    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87aeae47-0875-4102-8700-ae4b899cabbc-home" (OuterVolumeSpecName: "home") pod "87aeae47-0875-4102-8700-ae4b899cabbc" (UID: "87aeae47-0875-4102-8700-ae4b899cabbc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 16:21:44.807233 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:21:44.807204    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87aeae47-0875-4102-8700-ae4b899cabbc-kube-api-access-fvb7m" (OuterVolumeSpecName: "kube-api-access-fvb7m") pod "87aeae47-0875-4102-8700-ae4b899cabbc" (UID: "87aeae47-0875-4102-8700-ae4b899cabbc"). InnerVolumeSpecName "kube-api-access-fvb7m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 16:21:44.807343 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:21:44.807240    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87aeae47-0875-4102-8700-ae4b899cabbc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "87aeae47-0875-4102-8700-ae4b899cabbc" (UID: "87aeae47-0875-4102-8700-ae4b899cabbc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 16:21:44.807343 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:21:44.807247    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87aeae47-0875-4102-8700-ae4b899cabbc-dshm" (OuterVolumeSpecName: "dshm") pod "87aeae47-0875-4102-8700-ae4b899cabbc" (UID: "87aeae47-0875-4102-8700-ae4b899cabbc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 16:21:44.905761 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:21:44.905724    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87aeae47-0875-4102-8700-ae4b899cabbc-dshm\") on node \"ip-10-0-138-101.ec2.internal\" DevicePath \"\""
Jul 07 16:21:44.905761 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:21:44.905756    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87aeae47-0875-4102-8700-ae4b899cabbc-tls-certs\") on node \"ip-10-0-138-101.ec2.internal\" DevicePath \"\""
Jul 07 16:21:44.905761 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:21:44.905767    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87aeae47-0875-4102-8700-ae4b899cabbc-home\") on node \"ip-10-0-138-101.ec2.internal\" DevicePath \"\""
Jul 07 16:21:44.906042 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:21:44.905776    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvb7m\" (UniqueName: \"kubernetes.io/projected/87aeae47-0875-4102-8700-ae4b899cabbc-kube-api-access-fvb7m\") on node \"ip-10-0-138-101.ec2.internal\" DevicePath \"\""
Jul 07 16:21:45.541966 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:21:45.541923    2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-1cea82/auth-test-model-1cea82-kserve-5d754d768c-j7vnv" podUID="670a7180-ffe0-439a-ac0f-69a9db6b9b54" containerName="main" containerID="cri-o://189722d1d275e964c3100444e9fbb4d3204fc18e5693e23a0de291ce2ba11e8b" gracePeriod=45
Jul 07 16:21:45.585073 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:21:45.585046    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-94a996/auth-test-model-94a996-kserve-84649cc75-r5whk"
Jul 07 16:21:45.585248 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:21:45.585046    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-94a996/auth-test-model-94a996-kserve-84649cc75-r5whk" event={"ID":"87aeae47-0875-4102-8700-ae4b899cabbc","Type":"ContainerDied","Data":"ebbf31c6a8b8c872b89686cfd7d4a81f1b31c4f801125dade2413e8eff4bd428"}
Jul 07 16:21:45.585248 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:21:45.585172    2566 scope.go:117] "RemoveContainer" containerID="dffabf4d2faba1a6b88ac8d45095810e9a572e92697a121526269ffe97e83837"
Jul 07 16:21:45.627169 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:21:45.627130    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-94a996/auth-test-model-94a996-kserve-84649cc75-r5whk"]
Jul 07 16:21:45.633530 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:21:45.633499    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-94a996/auth-test-model-94a996-kserve-84649cc75-r5whk"]
Jul 07 16:21:45.806751 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:21:45.806683    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1cea82/auth-test-model-1cea82-kserve-5d754d768c-j7vnv"
Jul 07 16:21:45.914313 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:21:45.914272    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/670a7180-ffe0-439a-ac0f-69a9db6b9b54-model-cache\") pod \"670a7180-ffe0-439a-ac0f-69a9db6b9b54\" (UID: \"670a7180-ffe0-439a-ac0f-69a9db6b9b54\") "
Jul 07 16:21:45.914313 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:21:45.914310    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f7hwt\" (UniqueName: \"kubernetes.io/projected/670a7180-ffe0-439a-ac0f-69a9db6b9b54-kube-api-access-f7hwt\") pod \"670a7180-ffe0-439a-ac0f-69a9db6b9b54\" (UID: \"670a7180-ffe0-439a-ac0f-69a9db6b9b54\") "
Jul 07 16:21:45.914562 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:21:45.914336    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/670a7180-ffe0-439a-ac0f-69a9db6b9b54-tls-certs\") pod \"670a7180-ffe0-439a-ac0f-69a9db6b9b54\" (UID: \"670a7180-ffe0-439a-ac0f-69a9db6b9b54\") "
Jul 07 16:21:45.914562 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:21:45.914397    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/670a7180-ffe0-439a-ac0f-69a9db6b9b54-tmp-dir\") pod \"670a7180-ffe0-439a-ac0f-69a9db6b9b54\" (UID: \"670a7180-ffe0-439a-ac0f-69a9db6b9b54\") "
Jul 07 16:21:45.914562 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:21:45.914427    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/670a7180-ffe0-439a-ac0f-69a9db6b9b54-dshm\") pod \"670a7180-ffe0-439a-ac0f-69a9db6b9b54\" (UID: \"670a7180-ffe0-439a-ac0f-69a9db6b9b54\") "
Jul 07 16:21:45.914562 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:21:45.914454    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/670a7180-ffe0-439a-ac0f-69a9db6b9b54-home\") pod \"670a7180-ffe0-439a-ac0f-69a9db6b9b54\" (UID: \"670a7180-ffe0-439a-ac0f-69a9db6b9b54\") "
Jul 07 16:21:45.914759 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:21:45.914618    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/670a7180-ffe0-439a-ac0f-69a9db6b9b54-model-cache" (OuterVolumeSpecName: "model-cache") pod "670a7180-ffe0-439a-ac0f-69a9db6b9b54" (UID: "670a7180-ffe0-439a-ac0f-69a9db6b9b54"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 16:21:45.914759 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:21:45.914699    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/670a7180-ffe0-439a-ac0f-69a9db6b9b54-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "670a7180-ffe0-439a-ac0f-69a9db6b9b54" (UID: "670a7180-ffe0-439a-ac0f-69a9db6b9b54"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 16:21:45.914873 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:21:45.914808    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/670a7180-ffe0-439a-ac0f-69a9db6b9b54-home" (OuterVolumeSpecName: "home") pod "670a7180-ffe0-439a-ac0f-69a9db6b9b54" (UID: "670a7180-ffe0-439a-ac0f-69a9db6b9b54"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 16:21:45.916637 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:21:45.916613    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/670a7180-ffe0-439a-ac0f-69a9db6b9b54-dshm" (OuterVolumeSpecName: "dshm") pod "670a7180-ffe0-439a-ac0f-69a9db6b9b54" (UID: "670a7180-ffe0-439a-ac0f-69a9db6b9b54"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 16:21:45.916748 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:21:45.916672    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/670a7180-ffe0-439a-ac0f-69a9db6b9b54-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "670a7180-ffe0-439a-ac0f-69a9db6b9b54" (UID: "670a7180-ffe0-439a-ac0f-69a9db6b9b54"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 16:21:45.916748 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:21:45.916693    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/670a7180-ffe0-439a-ac0f-69a9db6b9b54-kube-api-access-f7hwt" (OuterVolumeSpecName: "kube-api-access-f7hwt") pod "670a7180-ffe0-439a-ac0f-69a9db6b9b54" (UID: "670a7180-ffe0-439a-ac0f-69a9db6b9b54"). InnerVolumeSpecName "kube-api-access-f7hwt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 16:21:46.015602 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:21:46.015562    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/670a7180-ffe0-439a-ac0f-69a9db6b9b54-model-cache\") on node \"ip-10-0-138-101.ec2.internal\" DevicePath \"\""
Jul 07 16:21:46.015602 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:21:46.015599    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7hwt\" (UniqueName: \"kubernetes.io/projected/670a7180-ffe0-439a-ac0f-69a9db6b9b54-kube-api-access-f7hwt\") on node \"ip-10-0-138-101.ec2.internal\" DevicePath \"\""
Jul 07 16:21:46.015808 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:21:46.015624    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/670a7180-ffe0-439a-ac0f-69a9db6b9b54-tls-certs\") on node \"ip-10-0-138-101.ec2.internal\" DevicePath \"\""
Jul 07 16:21:46.015808 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:21:46.015634    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/670a7180-ffe0-439a-ac0f-69a9db6b9b54-tmp-dir\") on node \"ip-10-0-138-101.ec2.internal\" DevicePath \"\""
Jul 07 16:21:46.015808 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:21:46.015642    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/670a7180-ffe0-439a-ac0f-69a9db6b9b54-dshm\") on node \"ip-10-0-138-101.ec2.internal\" DevicePath \"\""
Jul 07 16:21:46.015808 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:21:46.015649    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/670a7180-ffe0-439a-ac0f-69a9db6b9b54-home\") on node \"ip-10-0-138-101.ec2.internal\" DevicePath \"\""
Jul 07 16:21:46.479517 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:21:46.479471    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87aeae47-0875-4102-8700-ae4b899cabbc" path="/var/lib/kubelet/pods/87aeae47-0875-4102-8700-ae4b899cabbc/volumes"
Jul 07 16:21:46.591588 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:21:46.591550    2566 generic.go:356] "Generic (PLEG): container finished" podID="670a7180-ffe0-439a-ac0f-69a9db6b9b54" containerID="189722d1d275e964c3100444e9fbb4d3204fc18e5693e23a0de291ce2ba11e8b" exitCode=0
Jul 07 16:21:46.592048 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:21:46.591608    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1cea82/auth-test-model-1cea82-kserve-5d754d768c-j7vnv"
Jul 07 16:21:46.592048 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:21:46.591632    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1cea82/auth-test-model-1cea82-kserve-5d754d768c-j7vnv" event={"ID":"670a7180-ffe0-439a-ac0f-69a9db6b9b54","Type":"ContainerDied","Data":"189722d1d275e964c3100444e9fbb4d3204fc18e5693e23a0de291ce2ba11e8b"}
Jul 07 16:21:46.592048 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:21:46.591692    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1cea82/auth-test-model-1cea82-kserve-5d754d768c-j7vnv" event={"ID":"670a7180-ffe0-439a-ac0f-69a9db6b9b54","Type":"ContainerDied","Data":"982763aa47dab58c468502982fdc7f6a2dfcca70e07d95cdb68e300959d905ae"}
Jul 07 16:21:46.592048 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:21:46.591716    2566 scope.go:117] "RemoveContainer" containerID="189722d1d275e964c3100444e9fbb4d3204fc18e5693e23a0de291ce2ba11e8b"
Jul 07 16:21:46.600859 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:21:46.600837    2566 scope.go:117] "RemoveContainer" containerID="189722d1d275e964c3100444e9fbb4d3204fc18e5693e23a0de291ce2ba11e8b"
Jul 07 16:21:46.601158 ip-10-0-138-101 kubenswrapper[2566]: E0707 16:21:46.601138    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"189722d1d275e964c3100444e9fbb4d3204fc18e5693e23a0de291ce2ba11e8b\": container with ID starting with 189722d1d275e964c3100444e9fbb4d3204fc18e5693e23a0de291ce2ba11e8b not found: ID does not exist" containerID="189722d1d275e964c3100444e9fbb4d3204fc18e5693e23a0de291ce2ba11e8b"
Jul 07 16:21:46.601216 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:21:46.601167    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"189722d1d275e964c3100444e9fbb4d3204fc18e5693e23a0de291ce2ba11e8b"} err="failed to get container status \"189722d1d275e964c3100444e9fbb4d3204fc18e5693e23a0de291ce2ba11e8b\": rpc error: code = NotFound desc = could not find container \"189722d1d275e964c3100444e9fbb4d3204fc18e5693e23a0de291ce2ba11e8b\": container with ID starting with 189722d1d275e964c3100444e9fbb4d3204fc18e5693e23a0de291ce2ba11e8b not found: ID does not exist"
Jul 07 16:21:46.609572 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:21:46.609543    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-1cea82/auth-test-model-1cea82-kserve-5d754d768c-j7vnv"]
Jul 07 16:21:46.617805 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:21:46.617776    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-1cea82/auth-test-model-1cea82-kserve-5d754d768c-j7vnv"]
Jul 07 16:21:48.474876 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:21:48.474841    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="670a7180-ffe0-439a-ac0f-69a9db6b9b54" path="/var/lib/kubelet/pods/670a7180-ffe0-439a-ac0f-69a9db6b9b54/volumes"
Jul 07 16:22:01.746576 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:22:01.746543    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-94a996/sub-test-model-94a996-kserve-59ddc5c75f-g524w"]
Jul 07 16:22:01.747000 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:22:01.746921    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87aeae47-0875-4102-8700-ae4b899cabbc" containerName="main"
Jul 07 16:22:01.747000 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:22:01.746933    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="87aeae47-0875-4102-8700-ae4b899cabbc" containerName="main"
Jul 07 16:22:01.747000 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:22:01.746943    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="670a7180-ffe0-439a-ac0f-69a9db6b9b54" containerName="main"
Jul 07 16:22:01.747000 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:22:01.746949    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="670a7180-ffe0-439a-ac0f-69a9db6b9b54" containerName="main"
Jul 07 16:22:01.747131 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:22:01.747014    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="87aeae47-0875-4102-8700-ae4b899cabbc" containerName="main"
Jul 07 16:22:01.747131 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:22:01.747024    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="670a7180-ffe0-439a-ac0f-69a9db6b9b54" containerName="main"
Jul 07 16:22:01.751240 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:22:01.751223    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-94a996/sub-test-model-94a996-kserve-59ddc5c75f-g524w"
Jul 07 16:22:01.754686 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:22:01.754666    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-94a996\"/\"kube-root-ca.crt\""
Jul 07 16:22:01.754825 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:22:01.754666    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-94a996\"/\"default-dockercfg-2tt2z\""
Jul 07 16:22:01.754825 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:22:01.754668    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-94a996\"/\"openshift-service-ca.crt\""
Jul 07 16:22:01.754825 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:22:01.754718    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-94a996\"/\"sub-test-model-94a996-kserve-self-signed-certs\""
Jul 07 16:22:01.761159 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:22:01.761133    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-94a996/sub-test-model-94a996-kserve-59ddc5c75f-g524w"]
Jul 07 16:22:01.850405 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:22:01.850363    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99b7743f-d577-4df4-a5cf-b21817d224f2-home\") pod \"sub-test-model-94a996-kserve-59ddc5c75f-g524w\" (UID: \"99b7743f-d577-4df4-a5cf-b21817d224f2\") " pod="ai-tenant-e2e-shared-a-94a996/sub-test-model-94a996-kserve-59ddc5c75f-g524w"
Jul 07 16:22:01.850585 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:22:01.850430    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5kjp\" (UniqueName: \"kubernetes.io/projected/99b7743f-d577-4df4-a5cf-b21817d224f2-kube-api-access-m5kjp\") pod \"sub-test-model-94a996-kserve-59ddc5c75f-g524w\" (UID: \"99b7743f-d577-4df4-a5cf-b21817d224f2\") " pod="ai-tenant-e2e-shared-a-94a996/sub-test-model-94a996-kserve-59ddc5c75f-g524w"
Jul 07 16:22:01.850585 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:22:01.850472    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99b7743f-d577-4df4-a5cf-b21817d224f2-tls-certs\") pod \"sub-test-model-94a996-kserve-59ddc5c75f-g524w\" (UID: \"99b7743f-d577-4df4-a5cf-b21817d224f2\") " pod="ai-tenant-e2e-shared-a-94a996/sub-test-model-94a996-kserve-59ddc5c75f-g524w"
Jul 07 16:22:01.850585 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:22:01.850549    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99b7743f-d577-4df4-a5cf-b21817d224f2-tmp-dir\") pod \"sub-test-model-94a996-kserve-59ddc5c75f-g524w\" (UID: \"99b7743f-d577-4df4-a5cf-b21817d224f2\") " pod="ai-tenant-e2e-shared-a-94a996/sub-test-model-94a996-kserve-59ddc5c75f-g524w"
Jul 07 16:22:01.850705 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:22:01.850594    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99b7743f-d577-4df4-a5cf-b21817d224f2-dshm\") pod \"sub-test-model-94a996-kserve-59ddc5c75f-g524w\" (UID: \"99b7743f-d577-4df4-a5cf-b21817d224f2\") " pod="ai-tenant-e2e-shared-a-94a996/sub-test-model-94a996-kserve-59ddc5c75f-g524w"
Jul 07 16:22:01.850705 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:22:01.850618    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99b7743f-d577-4df4-a5cf-b21817d224f2-model-cache\") pod \"sub-test-model-94a996-kserve-59ddc5c75f-g524w\" (UID: \"99b7743f-d577-4df4-a5cf-b21817d224f2\") " pod="ai-tenant-e2e-shared-a-94a996/sub-test-model-94a996-kserve-59ddc5c75f-g524w"
Jul 07 16:22:01.951125 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:22:01.951082    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99b7743f-d577-4df4-a5cf-b21817d224f2-dshm\") pod \"sub-test-model-94a996-kserve-59ddc5c75f-g524w\" (UID: \"99b7743f-d577-4df4-a5cf-b21817d224f2\") " pod="ai-tenant-e2e-shared-a-94a996/sub-test-model-94a996-kserve-59ddc5c75f-g524w"
Jul 07 16:22:01.951125 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:22:01.951125    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99b7743f-d577-4df4-a5cf-b21817d224f2-model-cache\") pod \"sub-test-model-94a996-kserve-59ddc5c75f-g524w\" (UID: \"99b7743f-d577-4df4-a5cf-b21817d224f2\") " pod="ai-tenant-e2e-shared-a-94a996/sub-test-model-94a996-kserve-59ddc5c75f-g524w"
Jul 07 16:22:01.951393 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:22:01.951252    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99b7743f-d577-4df4-a5cf-b21817d224f2-home\") pod \"sub-test-model-94a996-kserve-59ddc5c75f-g524w\" (UID: \"99b7743f-d577-4df4-a5cf-b21817d224f2\") " pod="ai-tenant-e2e-shared-a-94a996/sub-test-model-94a996-kserve-59ddc5c75f-g524w"
Jul 07 16:22:01.951393 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:22:01.951290    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5kjp\" (UniqueName: \"kubernetes.io/projected/99b7743f-d577-4df4-a5cf-b21817d224f2-kube-api-access-m5kjp\") pod \"sub-test-model-94a996-kserve-59ddc5c75f-g524w\" (UID: \"99b7743f-d577-4df4-a5cf-b21817d224f2\") " pod="ai-tenant-e2e-shared-a-94a996/sub-test-model-94a996-kserve-59ddc5c75f-g524w"
Jul 07 16:22:01.951393 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:22:01.951325    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99b7743f-d577-4df4-a5cf-b21817d224f2-tls-certs\") pod \"sub-test-model-94a996-kserve-59ddc5c75f-g524w\" (UID: \"99b7743f-d577-4df4-a5cf-b21817d224f2\") " pod="ai-tenant-e2e-shared-a-94a996/sub-test-model-94a996-kserve-59ddc5c75f-g524w"
Jul 07 16:22:01.951393 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:22:01.951365    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99b7743f-d577-4df4-a5cf-b21817d224f2-tmp-dir\") pod \"sub-test-model-94a996-kserve-59ddc5c75f-g524w\" (UID: \"99b7743f-d577-4df4-a5cf-b21817d224f2\") " pod="ai-tenant-e2e-shared-a-94a996/sub-test-model-94a996-kserve-59ddc5c75f-g524w"
Jul 07 16:22:01.951615 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:22:01.951458    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99b7743f-d577-4df4-a5cf-b21817d224f2-model-cache\") pod \"sub-test-model-94a996-kserve-59ddc5c75f-g524w\" (UID: \"99b7743f-d577-4df4-a5cf-b21817d224f2\") " pod="ai-tenant-e2e-shared-a-94a996/sub-test-model-94a996-kserve-59ddc5c75f-g524w"
Jul 07 16:22:01.951669 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:22:01.951614    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99b7743f-d577-4df4-a5cf-b21817d224f2-home\") pod \"sub-test-model-94a996-kserve-59ddc5c75f-g524w\" (UID: \"99b7743f-d577-4df4-a5cf-b21817d224f2\") " pod="ai-tenant-e2e-shared-a-94a996/sub-test-model-94a996-kserve-59ddc5c75f-g524w"
Jul 07 16:22:01.951707 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:22:01.951666    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99b7743f-d577-4df4-a5cf-b21817d224f2-tmp-dir\") pod \"sub-test-model-94a996-kserve-59ddc5c75f-g524w\" (UID: \"99b7743f-d577-4df4-a5cf-b21817d224f2\") " pod="ai-tenant-e2e-shared-a-94a996/sub-test-model-94a996-kserve-59ddc5c75f-g524w"
Jul 07 16:22:01.953437 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:22:01.953406    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99b7743f-d577-4df4-a5cf-b21817d224f2-dshm\") pod \"sub-test-model-94a996-kserve-59ddc5c75f-g524w\" (UID: \"99b7743f-d577-4df4-a5cf-b21817d224f2\") " pod="ai-tenant-e2e-shared-a-94a996/sub-test-model-94a996-kserve-59ddc5c75f-g524w"
Jul 07 16:22:01.953762 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:22:01.953741    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99b7743f-d577-4df4-a5cf-b21817d224f2-tls-certs\") pod \"sub-test-model-94a996-kserve-59ddc5c75f-g524w\" (UID: \"99b7743f-d577-4df4-a5cf-b21817d224f2\") " pod="ai-tenant-e2e-shared-a-94a996/sub-test-model-94a996-kserve-59ddc5c75f-g524w"
Jul 07 16:22:01.960195 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:22:01.960160    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5kjp\" (UniqueName: \"kubernetes.io/projected/99b7743f-d577-4df4-a5cf-b21817d224f2-kube-api-access-m5kjp\") pod \"sub-test-model-94a996-kserve-59ddc5c75f-g524w\" (UID: \"99b7743f-d577-4df4-a5cf-b21817d224f2\") " pod="ai-tenant-e2e-shared-a-94a996/sub-test-model-94a996-kserve-59ddc5c75f-g524w"
Jul 07 16:22:02.060962 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:22:02.060846    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-94a996/sub-test-model-94a996-kserve-59ddc5c75f-g524w"
Jul 07 16:22:02.206817 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:22:02.206789    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-94a996/sub-test-model-94a996-kserve-59ddc5c75f-g524w"]
Jul 07 16:22:02.209537 ip-10-0-138-101 kubenswrapper[2566]: W0707 16:22:02.209511    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod99b7743f_d577_4df4_a5cf_b21817d224f2.slice/crio-a4c22d8733b75dda7f782f158244fd1b52c9e272de400cff4f29a167b77d11ff WatchSource:0}: Error finding container a4c22d8733b75dda7f782f158244fd1b52c9e272de400cff4f29a167b77d11ff: Status 404 returned error can't find the container with id a4c22d8733b75dda7f782f158244fd1b52c9e272de400cff4f29a167b77d11ff
Jul 07 16:22:02.678739 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:22:02.678702    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-94a996/sub-test-model-94a996-kserve-59ddc5c75f-g524w" event={"ID":"99b7743f-d577-4df4-a5cf-b21817d224f2","Type":"ContainerStarted","Data":"ee786013079b17f21c06ded002bdeb4a6940cc69a3d0f3f042fb8efcdd3adad0"}
Jul 07 16:22:02.678739 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:22:02.678747    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-94a996/sub-test-model-94a996-kserve-59ddc5c75f-g524w" event={"ID":"99b7743f-d577-4df4-a5cf-b21817d224f2","Type":"ContainerStarted","Data":"a4c22d8733b75dda7f782f158244fd1b52c9e272de400cff4f29a167b77d11ff"}
Jul 07 16:22:02.698686 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:22:02.698626    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-94a996/sub-test-model-94a996-kserve-59ddc5c75f-g524w" podStartSLOduration=1.698604155 podStartE2EDuration="1.698604155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:22:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:22:02.697978912 +0000 UTC m=+2276.786404771" watchObservedRunningTime="2026-07-07 16:22:02.698604155 +0000 UTC m=+2276.787029823"
Jul 07 16:22:05.634984 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:22:05.634941    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mk99s"]
Jul 07 16:22:12.061702 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:22:12.061655    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-94a996/sub-test-model-94a996-kserve-59ddc5c75f-g524w"
Jul 07 16:22:12.062139 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:22:12.061718    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-94a996/sub-test-model-94a996-kserve-59ddc5c75f-g524w"
Jul 07 16:22:12.078566 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:22:12.078534    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-94a996/sub-test-model-94a996-kserve-59ddc5c75f-g524w"
Jul 07 16:22:12.736163 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:22:12.736130    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-94a996/sub-test-model-94a996-kserve-59ddc5c75f-g524w"
Jul 07 16:22:33.042500 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:22:33.042469    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-1cea82/sub-test-model-1cea82-kserve-578bb975dc-j7zq8"]
Jul 07 16:22:33.046005 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:22:33.045986    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1cea82/sub-test-model-1cea82-kserve-578bb975dc-j7zq8"
Jul 07 16:22:33.048676 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:22:33.048650    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-1cea82\"/\"default-dockercfg-q6lbs\""
Jul 07 16:22:33.049517 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:22:33.049495    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-1cea82\"/\"sub-test-model-1cea82-kserve-self-signed-certs\""
Jul 07 16:22:33.049616 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:22:33.049513    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-1cea82\"/\"openshift-service-ca.crt\""
Jul 07 16:22:33.049616 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:22:33.049520    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-1cea82\"/\"kube-root-ca.crt\""
Jul 07 16:22:33.056249 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:22:33.056225    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-1cea82/sub-test-model-1cea82-kserve-578bb975dc-j7zq8"]
Jul 07 16:22:33.132701 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:22:33.132661    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8480c1f-02cf-44b9-9319-2f03cc4a440f-dshm\") pod \"sub-test-model-1cea82-kserve-578bb975dc-j7zq8\" (UID: \"b8480c1f-02cf-44b9-9319-2f03cc4a440f\") " pod="ai-tenant-e2e-shared-b-1cea82/sub-test-model-1cea82-kserve-578bb975dc-j7zq8"
Jul 07 16:22:33.132701 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:22:33.132707    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8480c1f-02cf-44b9-9319-2f03cc4a440f-model-cache\") pod \"sub-test-model-1cea82-kserve-578bb975dc-j7zq8\" (UID: \"b8480c1f-02cf-44b9-9319-2f03cc4a440f\") " pod="ai-tenant-e2e-shared-b-1cea82/sub-test-model-1cea82-kserve-578bb975dc-j7zq8"
Jul 07 16:22:33.132963 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:22:33.132730    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqp4x\" (UniqueName: \"kubernetes.io/projected/b8480c1f-02cf-44b9-9319-2f03cc4a440f-kube-api-access-mqp4x\") pod \"sub-test-model-1cea82-kserve-578bb975dc-j7zq8\" (UID: \"b8480c1f-02cf-44b9-9319-2f03cc4a440f\") " pod="ai-tenant-e2e-shared-b-1cea82/sub-test-model-1cea82-kserve-578bb975dc-j7zq8"
Jul 07 16:22:33.132963 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:22:33.132827    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8480c1f-02cf-44b9-9319-2f03cc4a440f-tls-certs\") pod \"sub-test-model-1cea82-kserve-578bb975dc-j7zq8\" (UID: \"b8480c1f-02cf-44b9-9319-2f03cc4a440f\") " pod="ai-tenant-e2e-shared-b-1cea82/sub-test-model-1cea82-kserve-578bb975dc-j7zq8"
Jul 07 16:22:33.132963 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:22:33.132896    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8480c1f-02cf-44b9-9319-2f03cc4a440f-tmp-dir\") pod \"sub-test-model-1cea82-kserve-578bb975dc-j7zq8\" (UID: \"b8480c1f-02cf-44b9-9319-2f03cc4a440f\") " pod="ai-tenant-e2e-shared-b-1cea82/sub-test-model-1cea82-kserve-578bb975dc-j7zq8"
Jul 07 16:22:33.133111 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:22:33.133018    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8480c1f-02cf-44b9-9319-2f03cc4a440f-home\") pod \"sub-test-model-1cea82-kserve-578bb975dc-j7zq8\" (UID: \"b8480c1f-02cf-44b9-9319-2f03cc4a440f\") " pod="ai-tenant-e2e-shared-b-1cea82/sub-test-model-1cea82-kserve-578bb975dc-j7zq8"
Jul 07 16:22:33.234215 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:22:33.234165    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8480c1f-02cf-44b9-9319-2f03cc4a440f-home\") pod \"sub-test-model-1cea82-kserve-578bb975dc-j7zq8\" (UID: \"b8480c1f-02cf-44b9-9319-2f03cc4a440f\") " pod="ai-tenant-e2e-shared-b-1cea82/sub-test-model-1cea82-kserve-578bb975dc-j7zq8"
Jul 07 16:22:33.234215 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:22:33.234223    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8480c1f-02cf-44b9-9319-2f03cc4a440f-dshm\") pod \"sub-test-model-1cea82-kserve-578bb975dc-j7zq8\" (UID: \"b8480c1f-02cf-44b9-9319-2f03cc4a440f\") " pod="ai-tenant-e2e-shared-b-1cea82/sub-test-model-1cea82-kserve-578bb975dc-j7zq8"
Jul 07 16:22:33.234487 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:22:33.234247    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8480c1f-02cf-44b9-9319-2f03cc4a440f-model-cache\") pod \"sub-test-model-1cea82-kserve-578bb975dc-j7zq8\" (UID: \"b8480c1f-02cf-44b9-9319-2f03cc4a440f\") " pod="ai-tenant-e2e-shared-b-1cea82/sub-test-model-1cea82-kserve-578bb975dc-j7zq8"
Jul 07 16:22:33.234487 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:22:33.234269    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqp4x\" (UniqueName: \"kubernetes.io/projected/b8480c1f-02cf-44b9-9319-2f03cc4a440f-kube-api-access-mqp4x\") pod \"sub-test-model-1cea82-kserve-578bb975dc-j7zq8\" (UID: \"b8480c1f-02cf-44b9-9319-2f03cc4a440f\") " pod="ai-tenant-e2e-shared-b-1cea82/sub-test-model-1cea82-kserve-578bb975dc-j7zq8"
Jul 07 16:22:33.234487 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:22:33.234325    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8480c1f-02cf-44b9-9319-2f03cc4a440f-tls-certs\") pod \"sub-test-model-1cea82-kserve-578bb975dc-j7zq8\" (UID: \"b8480c1f-02cf-44b9-9319-2f03cc4a440f\") " pod="ai-tenant-e2e-shared-b-1cea82/sub-test-model-1cea82-kserve-578bb975dc-j7zq8"
Jul 07 16:22:33.234487 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:22:33.234380    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8480c1f-02cf-44b9-9319-2f03cc4a440f-tmp-dir\") pod \"sub-test-model-1cea82-kserve-578bb975dc-j7zq8\" (UID: \"b8480c1f-02cf-44b9-9319-2f03cc4a440f\") " pod="ai-tenant-e2e-shared-b-1cea82/sub-test-model-1cea82-kserve-578bb975dc-j7zq8"
Jul 07 16:22:33.234697 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:22:33.234573    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8480c1f-02cf-44b9-9319-2f03cc4a440f-home\") pod \"sub-test-model-1cea82-kserve-578bb975dc-j7zq8\" (UID: \"b8480c1f-02cf-44b9-9319-2f03cc4a440f\") " pod="ai-tenant-e2e-shared-b-1cea82/sub-test-model-1cea82-kserve-578bb975dc-j7zq8"
Jul 07 16:22:33.234758 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:22:33.234699    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8480c1f-02cf-44b9-9319-2f03cc4a440f-model-cache\") pod \"sub-test-model-1cea82-kserve-578bb975dc-j7zq8\" (UID: \"b8480c1f-02cf-44b9-9319-2f03cc4a440f\") " pod="ai-tenant-e2e-shared-b-1cea82/sub-test-model-1cea82-kserve-578bb975dc-j7zq8"
Jul 07 16:22:33.234758 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:22:33.234745    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8480c1f-02cf-44b9-9319-2f03cc4a440f-tmp-dir\") pod \"sub-test-model-1cea82-kserve-578bb975dc-j7zq8\" (UID: \"b8480c1f-02cf-44b9-9319-2f03cc4a440f\") " pod="ai-tenant-e2e-shared-b-1cea82/sub-test-model-1cea82-kserve-578bb975dc-j7zq8"
Jul 07 16:22:33.236622 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:22:33.236589    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8480c1f-02cf-44b9-9319-2f03cc4a440f-dshm\") pod \"sub-test-model-1cea82-kserve-578bb975dc-j7zq8\" (UID: \"b8480c1f-02cf-44b9-9319-2f03cc4a440f\") " pod="ai-tenant-e2e-shared-b-1cea82/sub-test-model-1cea82-kserve-578bb975dc-j7zq8"
Jul 07 16:22:33.236819 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:22:33.236804    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8480c1f-02cf-44b9-9319-2f03cc4a440f-tls-certs\") pod \"sub-test-model-1cea82-kserve-578bb975dc-j7zq8\" (UID: \"b8480c1f-02cf-44b9-9319-2f03cc4a440f\") " pod="ai-tenant-e2e-shared-b-1cea82/sub-test-model-1cea82-kserve-578bb975dc-j7zq8"
Jul 07 16:22:33.247124 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:22:33.247097    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqp4x\" (UniqueName: \"kubernetes.io/projected/b8480c1f-02cf-44b9-9319-2f03cc4a440f-kube-api-access-mqp4x\") pod \"sub-test-model-1cea82-kserve-578bb975dc-j7zq8\" (UID: \"b8480c1f-02cf-44b9-9319-2f03cc4a440f\") " pod="ai-tenant-e2e-shared-b-1cea82/sub-test-model-1cea82-kserve-578bb975dc-j7zq8"
Jul 07 16:22:33.356022 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:22:33.355916    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1cea82/sub-test-model-1cea82-kserve-578bb975dc-j7zq8"
Jul 07 16:22:33.507229 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:22:33.507195    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-1cea82/sub-test-model-1cea82-kserve-578bb975dc-j7zq8"]
Jul 07 16:22:33.510299 ip-10-0-138-101 kubenswrapper[2566]: W0707 16:22:33.510269    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb8480c1f_02cf_44b9_9319_2f03cc4a440f.slice/crio-9451d868a3efd1fa5661fa9abe80cadf2f05c76108078caada5e74d8bd3939ec WatchSource:0}: Error finding container 9451d868a3efd1fa5661fa9abe80cadf2f05c76108078caada5e74d8bd3939ec: Status 404 returned error can't find the container with id 9451d868a3efd1fa5661fa9abe80cadf2f05c76108078caada5e74d8bd3939ec
Jul 07 16:22:33.824626 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:22:33.824585    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1cea82/sub-test-model-1cea82-kserve-578bb975dc-j7zq8" event={"ID":"b8480c1f-02cf-44b9-9319-2f03cc4a440f","Type":"ContainerStarted","Data":"1b970636f5ba992edd42eb85094bfc9e94b4c86c7641418db973a87018ffc1a8"}
Jul 07 16:22:33.824626 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:22:33.824633    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1cea82/sub-test-model-1cea82-kserve-578bb975dc-j7zq8" event={"ID":"b8480c1f-02cf-44b9-9319-2f03cc4a440f","Type":"ContainerStarted","Data":"9451d868a3efd1fa5661fa9abe80cadf2f05c76108078caada5e74d8bd3939ec"}
Jul 07 16:22:33.846375 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:22:33.846314    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-1cea82/sub-test-model-1cea82-kserve-578bb975dc-j7zq8" podStartSLOduration=0.846296602 podStartE2EDuration="846.296602ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:22:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:22:33.844975708 +0000 UTC m=+2307.933401378" watchObservedRunningTime="2026-07-07 16:22:33.846296602 +0000 UTC m=+2307.934722271"
Jul 07 16:22:37.438875 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:22:37.438839    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mk99s"]
Jul 07 16:22:43.356649 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:22:43.356611    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-1cea82/sub-test-model-1cea82-kserve-578bb975dc-j7zq8"
Jul 07 16:22:43.357096 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:22:43.356697    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-1cea82/sub-test-model-1cea82-kserve-578bb975dc-j7zq8"
Jul 07 16:22:43.375068 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:22:43.375033    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-1cea82/sub-test-model-1cea82-kserve-578bb975dc-j7zq8"
Jul 07 16:22:43.885518 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:22:43.885487    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-1cea82/sub-test-model-1cea82-kserve-578bb975dc-j7zq8"
Jul 07 16:22:56.333758 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:22:56.333723    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mk99s"]
Jul 07 16:23:09.741857 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:09.741822    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-94a996/sub-test-model-94a996-kserve-59ddc5c75f-g524w"]
Jul 07 16:23:10.718959 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:10.718924    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-1cea82/sub-test-model-1cea82-kserve-578bb975dc-j7zq8"]
Jul 07 16:23:15.139405 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:15.139368    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mk99s"]
Jul 07 16:23:24.761763 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:24.761704    2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-94a996/sub-test-model-94a996-kserve-59ddc5c75f-g524w" podUID="99b7743f-d577-4df4-a5cf-b21817d224f2" containerName="main" containerID="cri-o://ee786013079b17f21c06ded002bdeb4a6940cc69a3d0f3f042fb8efcdd3adad0" gracePeriod=45
Jul 07 16:23:25.023284 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:25.023215    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-94a996/sub-test-model-94a996-kserve-59ddc5c75f-g524w"
Jul 07 16:23:25.068492 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:25.068453    2566 generic.go:356] "Generic (PLEG): container finished" podID="99b7743f-d577-4df4-a5cf-b21817d224f2" containerID="ee786013079b17f21c06ded002bdeb4a6940cc69a3d0f3f042fb8efcdd3adad0" exitCode=0
Jul 07 16:23:25.068492 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:25.068496    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-94a996/sub-test-model-94a996-kserve-59ddc5c75f-g524w"
Jul 07 16:23:25.068740 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:25.068544    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-94a996/sub-test-model-94a996-kserve-59ddc5c75f-g524w" event={"ID":"99b7743f-d577-4df4-a5cf-b21817d224f2","Type":"ContainerDied","Data":"ee786013079b17f21c06ded002bdeb4a6940cc69a3d0f3f042fb8efcdd3adad0"}
Jul 07 16:23:25.068740 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:25.068594    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-94a996/sub-test-model-94a996-kserve-59ddc5c75f-g524w" event={"ID":"99b7743f-d577-4df4-a5cf-b21817d224f2","Type":"ContainerDied","Data":"a4c22d8733b75dda7f782f158244fd1b52c9e272de400cff4f29a167b77d11ff"}
Jul 07 16:23:25.068740 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:25.068616    2566 scope.go:117] "RemoveContainer" containerID="ee786013079b17f21c06ded002bdeb4a6940cc69a3d0f3f042fb8efcdd3adad0"
Jul 07 16:23:25.078356 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:25.078334    2566 scope.go:117] "RemoveContainer" containerID="ee786013079b17f21c06ded002bdeb4a6940cc69a3d0f3f042fb8efcdd3adad0"
Jul 07 16:23:25.078656 ip-10-0-138-101 kubenswrapper[2566]: E0707 16:23:25.078637    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee786013079b17f21c06ded002bdeb4a6940cc69a3d0f3f042fb8efcdd3adad0\": container with ID starting with ee786013079b17f21c06ded002bdeb4a6940cc69a3d0f3f042fb8efcdd3adad0 not found: ID does not exist" containerID="ee786013079b17f21c06ded002bdeb4a6940cc69a3d0f3f042fb8efcdd3adad0"
Jul 07 16:23:25.078754 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:25.078671    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee786013079b17f21c06ded002bdeb4a6940cc69a3d0f3f042fb8efcdd3adad0"} err="failed to get container status \"ee786013079b17f21c06ded002bdeb4a6940cc69a3d0f3f042fb8efcdd3adad0\": rpc error: code = NotFound desc = could not find container \"ee786013079b17f21c06ded002bdeb4a6940cc69a3d0f3f042fb8efcdd3adad0\": container with ID starting with ee786013079b17f21c06ded002bdeb4a6940cc69a3d0f3f042fb8efcdd3adad0 not found: ID does not exist"
Jul 07 16:23:25.115153 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:25.115115    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m5kjp\" (UniqueName: \"kubernetes.io/projected/99b7743f-d577-4df4-a5cf-b21817d224f2-kube-api-access-m5kjp\") pod \"99b7743f-d577-4df4-a5cf-b21817d224f2\" (UID: \"99b7743f-d577-4df4-a5cf-b21817d224f2\") "
Jul 07 16:23:25.115335 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:25.115176    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99b7743f-d577-4df4-a5cf-b21817d224f2-model-cache\") pod \"99b7743f-d577-4df4-a5cf-b21817d224f2\" (UID: \"99b7743f-d577-4df4-a5cf-b21817d224f2\") "
Jul 07 16:23:25.115335 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:25.115206    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99b7743f-d577-4df4-a5cf-b21817d224f2-tls-certs\") pod \"99b7743f-d577-4df4-a5cf-b21817d224f2\" (UID: \"99b7743f-d577-4df4-a5cf-b21817d224f2\") "
Jul 07 16:23:25.115335 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:25.115231    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99b7743f-d577-4df4-a5cf-b21817d224f2-dshm\") pod \"99b7743f-d577-4df4-a5cf-b21817d224f2\" (UID: \"99b7743f-d577-4df4-a5cf-b21817d224f2\") "
Jul 07 16:23:25.115504 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:25.115390    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99b7743f-d577-4df4-a5cf-b21817d224f2-home\") pod \"99b7743f-d577-4df4-a5cf-b21817d224f2\" (UID: \"99b7743f-d577-4df4-a5cf-b21817d224f2\") "
Jul 07 16:23:25.115504 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:25.115490    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99b7743f-d577-4df4-a5cf-b21817d224f2-model-cache" (OuterVolumeSpecName: "model-cache") pod "99b7743f-d577-4df4-a5cf-b21817d224f2" (UID: "99b7743f-d577-4df4-a5cf-b21817d224f2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 16:23:25.116174 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:25.116087    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99b7743f-d577-4df4-a5cf-b21817d224f2-tmp-dir\") pod \"99b7743f-d577-4df4-a5cf-b21817d224f2\" (UID: \"99b7743f-d577-4df4-a5cf-b21817d224f2\") "
Jul 07 16:23:25.116174 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:25.116105    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99b7743f-d577-4df4-a5cf-b21817d224f2-home" (OuterVolumeSpecName: "home") pod "99b7743f-d577-4df4-a5cf-b21817d224f2" (UID: "99b7743f-d577-4df4-a5cf-b21817d224f2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 16:23:25.116377 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:25.116356    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99b7743f-d577-4df4-a5cf-b21817d224f2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "99b7743f-d577-4df4-a5cf-b21817d224f2" (UID: "99b7743f-d577-4df4-a5cf-b21817d224f2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 16:23:25.117775 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:25.117690    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99b7743f-d577-4df4-a5cf-b21817d224f2-tmp-dir\") on node \"ip-10-0-138-101.ec2.internal\" DevicePath \"\""
Jul 07 16:23:25.117775 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:25.117717    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99b7743f-d577-4df4-a5cf-b21817d224f2-model-cache\") on node \"ip-10-0-138-101.ec2.internal\" DevicePath \"\""
Jul 07 16:23:25.117775 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:25.117731    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99b7743f-d577-4df4-a5cf-b21817d224f2-home\") on node \"ip-10-0-138-101.ec2.internal\" DevicePath \"\""
Jul 07 16:23:25.123398 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:25.118032    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99b7743f-d577-4df4-a5cf-b21817d224f2-dshm" (OuterVolumeSpecName: "dshm") pod "99b7743f-d577-4df4-a5cf-b21817d224f2" (UID: "99b7743f-d577-4df4-a5cf-b21817d224f2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 16:23:25.123775 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:25.123750    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99b7743f-d577-4df4-a5cf-b21817d224f2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "99b7743f-d577-4df4-a5cf-b21817d224f2" (UID: "99b7743f-d577-4df4-a5cf-b21817d224f2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 16:23:25.123775 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:25.123765    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99b7743f-d577-4df4-a5cf-b21817d224f2-kube-api-access-m5kjp" (OuterVolumeSpecName: "kube-api-access-m5kjp") pod "99b7743f-d577-4df4-a5cf-b21817d224f2" (UID: "99b7743f-d577-4df4-a5cf-b21817d224f2"). InnerVolumeSpecName "kube-api-access-m5kjp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 16:23:25.218710 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:25.218668    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99b7743f-d577-4df4-a5cf-b21817d224f2-tls-certs\") on node \"ip-10-0-138-101.ec2.internal\" DevicePath \"\""
Jul 07 16:23:25.218710 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:25.218704    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99b7743f-d577-4df4-a5cf-b21817d224f2-dshm\") on node \"ip-10-0-138-101.ec2.internal\" DevicePath \"\""
Jul 07 16:23:25.218710 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:25.218715    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5kjp\" (UniqueName: \"kubernetes.io/projected/99b7743f-d577-4df4-a5cf-b21817d224f2-kube-api-access-m5kjp\") on node \"ip-10-0-138-101.ec2.internal\" DevicePath \"\""
Jul 07 16:23:25.392276 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:25.392233    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-94a996/sub-test-model-94a996-kserve-59ddc5c75f-g524w"]
Jul 07 16:23:25.397735 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:25.397702    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-94a996/sub-test-model-94a996-kserve-59ddc5c75f-g524w"]
Jul 07 16:23:25.738810 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:25.738772    2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-1cea82/sub-test-model-1cea82-kserve-578bb975dc-j7zq8" podUID="b8480c1f-02cf-44b9-9319-2f03cc4a440f" containerName="main" containerID="cri-o://1b970636f5ba992edd42eb85094bfc9e94b4c86c7641418db973a87018ffc1a8" gracePeriod=45
Jul 07 16:23:26.000218 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:26.000152    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1cea82/sub-test-model-1cea82-kserve-578bb975dc-j7zq8"
Jul 07 16:23:26.074475 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:26.074441    2566 generic.go:356] "Generic (PLEG): container finished" podID="b8480c1f-02cf-44b9-9319-2f03cc4a440f" containerID="1b970636f5ba992edd42eb85094bfc9e94b4c86c7641418db973a87018ffc1a8" exitCode=0
Jul 07 16:23:26.074667 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:26.074487    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1cea82/sub-test-model-1cea82-kserve-578bb975dc-j7zq8"
Jul 07 16:23:26.074667 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:26.074525    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1cea82/sub-test-model-1cea82-kserve-578bb975dc-j7zq8" event={"ID":"b8480c1f-02cf-44b9-9319-2f03cc4a440f","Type":"ContainerDied","Data":"1b970636f5ba992edd42eb85094bfc9e94b4c86c7641418db973a87018ffc1a8"}
Jul 07 16:23:26.074667 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:26.074567    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1cea82/sub-test-model-1cea82-kserve-578bb975dc-j7zq8" event={"ID":"b8480c1f-02cf-44b9-9319-2f03cc4a440f","Type":"ContainerDied","Data":"9451d868a3efd1fa5661fa9abe80cadf2f05c76108078caada5e74d8bd3939ec"}
Jul 07 16:23:26.074667 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:26.074591    2566 scope.go:117] "RemoveContainer" containerID="1b970636f5ba992edd42eb85094bfc9e94b4c86c7641418db973a87018ffc1a8"
Jul 07 16:23:26.085080 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:26.084833    2566 scope.go:117] "RemoveContainer" containerID="1b970636f5ba992edd42eb85094bfc9e94b4c86c7641418db973a87018ffc1a8"
Jul 07 16:23:26.085520 ip-10-0-138-101 kubenswrapper[2566]: E0707 16:23:26.085501    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b970636f5ba992edd42eb85094bfc9e94b4c86c7641418db973a87018ffc1a8\": container with ID starting with 1b970636f5ba992edd42eb85094bfc9e94b4c86c7641418db973a87018ffc1a8 not found: ID does not exist" containerID="1b970636f5ba992edd42eb85094bfc9e94b4c86c7641418db973a87018ffc1a8"
Jul 07 16:23:26.085590 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:26.085532    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b970636f5ba992edd42eb85094bfc9e94b4c86c7641418db973a87018ffc1a8"} err="failed to get container status \"1b970636f5ba992edd42eb85094bfc9e94b4c86c7641418db973a87018ffc1a8\": rpc error: code = NotFound desc = could not find container \"1b970636f5ba992edd42eb85094bfc9e94b4c86c7641418db973a87018ffc1a8\": container with ID starting with 1b970636f5ba992edd42eb85094bfc9e94b4c86c7641418db973a87018ffc1a8 not found: ID does not exist"
Jul 07 16:23:26.126977 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:26.126946    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8480c1f-02cf-44b9-9319-2f03cc4a440f-home\") pod \"b8480c1f-02cf-44b9-9319-2f03cc4a440f\" (UID: \"b8480c1f-02cf-44b9-9319-2f03cc4a440f\") "
Jul 07 16:23:26.126977 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:26.126986    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8480c1f-02cf-44b9-9319-2f03cc4a440f-model-cache\") pod \"b8480c1f-02cf-44b9-9319-2f03cc4a440f\" (UID: \"b8480c1f-02cf-44b9-9319-2f03cc4a440f\") "
Jul 07 16:23:26.127192 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:26.127021    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8480c1f-02cf-44b9-9319-2f03cc4a440f-dshm\") pod \"b8480c1f-02cf-44b9-9319-2f03cc4a440f\" (UID: \"b8480c1f-02cf-44b9-9319-2f03cc4a440f\") "
Jul 07 16:23:26.127192 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:26.127072    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8480c1f-02cf-44b9-9319-2f03cc4a440f-tmp-dir\") pod \"b8480c1f-02cf-44b9-9319-2f03cc4a440f\" (UID: \"b8480c1f-02cf-44b9-9319-2f03cc4a440f\") "
Jul 07 16:23:26.127192 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:26.127142    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8480c1f-02cf-44b9-9319-2f03cc4a440f-tls-certs\") pod \"b8480c1f-02cf-44b9-9319-2f03cc4a440f\" (UID: \"b8480c1f-02cf-44b9-9319-2f03cc4a440f\") "
Jul 07 16:23:26.127192 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:26.127172    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mqp4x\" (UniqueName: \"kubernetes.io/projected/b8480c1f-02cf-44b9-9319-2f03cc4a440f-kube-api-access-mqp4x\") pod \"b8480c1f-02cf-44b9-9319-2f03cc4a440f\" (UID: \"b8480c1f-02cf-44b9-9319-2f03cc4a440f\") "
Jul 07 16:23:26.127398 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:26.127270    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8480c1f-02cf-44b9-9319-2f03cc4a440f-home" (OuterVolumeSpecName: "home") pod "b8480c1f-02cf-44b9-9319-2f03cc4a440f" (UID: "b8480c1f-02cf-44b9-9319-2f03cc4a440f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 16:23:26.127398 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:26.127281    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8480c1f-02cf-44b9-9319-2f03cc4a440f-model-cache" (OuterVolumeSpecName: "model-cache") pod "b8480c1f-02cf-44b9-9319-2f03cc4a440f" (UID: "b8480c1f-02cf-44b9-9319-2f03cc4a440f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 16:23:26.127509 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:26.127395    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8480c1f-02cf-44b9-9319-2f03cc4a440f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b8480c1f-02cf-44b9-9319-2f03cc4a440f" (UID: "b8480c1f-02cf-44b9-9319-2f03cc4a440f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 16:23:26.127509 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:26.127484    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8480c1f-02cf-44b9-9319-2f03cc4a440f-tmp-dir\") on node \"ip-10-0-138-101.ec2.internal\" DevicePath \"\""
Jul 07 16:23:26.127509 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:26.127505    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8480c1f-02cf-44b9-9319-2f03cc4a440f-home\") on node \"ip-10-0-138-101.ec2.internal\" DevicePath \"\""
Jul 07 16:23:26.127643 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:26.127518    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8480c1f-02cf-44b9-9319-2f03cc4a440f-model-cache\") on node \"ip-10-0-138-101.ec2.internal\" DevicePath \"\""
Jul 07 16:23:26.129276 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:26.129251    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8480c1f-02cf-44b9-9319-2f03cc4a440f-dshm" (OuterVolumeSpecName: "dshm") pod "b8480c1f-02cf-44b9-9319-2f03cc4a440f" (UID: "b8480c1f-02cf-44b9-9319-2f03cc4a440f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 16:23:26.129388 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:26.129296    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8480c1f-02cf-44b9-9319-2f03cc4a440f-kube-api-access-mqp4x" (OuterVolumeSpecName: "kube-api-access-mqp4x") pod "b8480c1f-02cf-44b9-9319-2f03cc4a440f" (UID: "b8480c1f-02cf-44b9-9319-2f03cc4a440f"). InnerVolumeSpecName "kube-api-access-mqp4x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 16:23:26.129388 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:26.129302    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8480c1f-02cf-44b9-9319-2f03cc4a440f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b8480c1f-02cf-44b9-9319-2f03cc4a440f" (UID: "b8480c1f-02cf-44b9-9319-2f03cc4a440f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 16:23:26.228385 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:26.228341    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8480c1f-02cf-44b9-9319-2f03cc4a440f-tls-certs\") on node \"ip-10-0-138-101.ec2.internal\" DevicePath \"\""
Jul 07 16:23:26.228385 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:26.228377    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqp4x\" (UniqueName: \"kubernetes.io/projected/b8480c1f-02cf-44b9-9319-2f03cc4a440f-kube-api-access-mqp4x\") on node \"ip-10-0-138-101.ec2.internal\" DevicePath \"\""
Jul 07 16:23:26.228385 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:26.228387    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8480c1f-02cf-44b9-9319-2f03cc4a440f-dshm\") on node \"ip-10-0-138-101.ec2.internal\" DevicePath \"\""
Jul 07 16:23:26.398057 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:26.398019    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-1cea82/sub-test-model-1cea82-kserve-578bb975dc-j7zq8"]
Jul 07 16:23:26.402674 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:26.402645    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-1cea82/sub-test-model-1cea82-kserve-578bb975dc-j7zq8"]
Jul 07 16:23:26.475283 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:26.475251    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99b7743f-d577-4df4-a5cf-b21817d224f2" path="/var/lib/kubelet/pods/99b7743f-d577-4df4-a5cf-b21817d224f2/volumes"
Jul 07 16:23:26.475659 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:26.475646    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8480c1f-02cf-44b9-9319-2f03cc4a440f" path="/var/lib/kubelet/pods/b8480c1f-02cf-44b9-9319-2f03cc4a440f/volumes"
Jul 07 16:23:27.740679 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:27.740647    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-94a996/rate-test-model-94a996-kserve-84f5bc78b7-thzvn"]
Jul 07 16:23:27.741148 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:27.741025    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8480c1f-02cf-44b9-9319-2f03cc4a440f" containerName="main"
Jul 07 16:23:27.741148 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:27.741039    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8480c1f-02cf-44b9-9319-2f03cc4a440f" containerName="main"
Jul 07 16:23:27.741148 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:27.741057    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99b7743f-d577-4df4-a5cf-b21817d224f2" containerName="main"
Jul 07 16:23:27.741148 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:27.741063    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="99b7743f-d577-4df4-a5cf-b21817d224f2" containerName="main"
Jul 07 16:23:27.741148 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:27.741120    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="99b7743f-d577-4df4-a5cf-b21817d224f2" containerName="main"
Jul 07 16:23:27.741148 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:27.741128    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8480c1f-02cf-44b9-9319-2f03cc4a440f" containerName="main"
Jul 07 16:23:27.745310 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:27.745290    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-94a996/rate-test-model-94a996-kserve-84f5bc78b7-thzvn"
Jul 07 16:23:27.748965 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:27.748941    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-94a996\"/\"default-dockercfg-2tt2z\""
Jul 07 16:23:27.748965 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:27.748955    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-94a996\"/\"openshift-service-ca.crt\""
Jul 07 16:23:27.749183 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:27.748971    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-94a996\"/\"rate-test-model-94a996-kserve-self-signed-certs\""
Jul 07 16:23:27.749183 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:27.749053    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-94a996\"/\"kube-root-ca.crt\""
Jul 07 16:23:27.755039 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:27.755015    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-94a996/rate-test-model-94a996-kserve-84f5bc78b7-thzvn"]
Jul 07 16:23:27.841995 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:27.841948    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwp9b\" (UniqueName: \"kubernetes.io/projected/72882acd-bf9d-437a-8d6d-178506f8f24a-kube-api-access-pwp9b\") pod \"rate-test-model-94a996-kserve-84f5bc78b7-thzvn\" (UID: \"72882acd-bf9d-437a-8d6d-178506f8f24a\") " pod="ai-tenant-e2e-shared-a-94a996/rate-test-model-94a996-kserve-84f5bc78b7-thzvn"
Jul 07 16:23:27.842196 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:27.842033    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72882acd-bf9d-437a-8d6d-178506f8f24a-home\") pod \"rate-test-model-94a996-kserve-84f5bc78b7-thzvn\" (UID: \"72882acd-bf9d-437a-8d6d-178506f8f24a\") " pod="ai-tenant-e2e-shared-a-94a996/rate-test-model-94a996-kserve-84f5bc78b7-thzvn"
Jul 07 16:23:27.842196 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:27.842076    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72882acd-bf9d-437a-8d6d-178506f8f24a-tmp-dir\") pod \"rate-test-model-94a996-kserve-84f5bc78b7-thzvn\" (UID: \"72882acd-bf9d-437a-8d6d-178506f8f24a\") " pod="ai-tenant-e2e-shared-a-94a996/rate-test-model-94a996-kserve-84f5bc78b7-thzvn"
Jul 07 16:23:27.842196 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:27.842131    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72882acd-bf9d-437a-8d6d-178506f8f24a-tls-certs\") pod \"rate-test-model-94a996-kserve-84f5bc78b7-thzvn\" (UID: \"72882acd-bf9d-437a-8d6d-178506f8f24a\") " pod="ai-tenant-e2e-shared-a-94a996/rate-test-model-94a996-kserve-84f5bc78b7-thzvn"
Jul 07 16:23:27.842196 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:27.842166    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72882acd-bf9d-437a-8d6d-178506f8f24a-dshm\") pod \"rate-test-model-94a996-kserve-84f5bc78b7-thzvn\" (UID: \"72882acd-bf9d-437a-8d6d-178506f8f24a\") " pod="ai-tenant-e2e-shared-a-94a996/rate-test-model-94a996-kserve-84f5bc78b7-thzvn"
Jul 07 16:23:27.842354 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:27.842219    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72882acd-bf9d-437a-8d6d-178506f8f24a-model-cache\") pod \"rate-test-model-94a996-kserve-84f5bc78b7-thzvn\" (UID: \"72882acd-bf9d-437a-8d6d-178506f8f24a\") " pod="ai-tenant-e2e-shared-a-94a996/rate-test-model-94a996-kserve-84f5bc78b7-thzvn"
Jul 07 16:23:27.943281 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:27.943240    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwp9b\" (UniqueName: \"kubernetes.io/projected/72882acd-bf9d-437a-8d6d-178506f8f24a-kube-api-access-pwp9b\") pod \"rate-test-model-94a996-kserve-84f5bc78b7-thzvn\" (UID: \"72882acd-bf9d-437a-8d6d-178506f8f24a\") " pod="ai-tenant-e2e-shared-a-94a996/rate-test-model-94a996-kserve-84f5bc78b7-thzvn"
Jul 07 16:23:27.943281 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:27.943287    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72882acd-bf9d-437a-8d6d-178506f8f24a-home\") pod \"rate-test-model-94a996-kserve-84f5bc78b7-thzvn\" (UID: \"72882acd-bf9d-437a-8d6d-178506f8f24a\") " pod="ai-tenant-e2e-shared-a-94a996/rate-test-model-94a996-kserve-84f5bc78b7-thzvn"
Jul 07 16:23:27.943569 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:27.943426    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72882acd-bf9d-437a-8d6d-178506f8f24a-tmp-dir\") pod \"rate-test-model-94a996-kserve-84f5bc78b7-thzvn\" (UID: \"72882acd-bf9d-437a-8d6d-178506f8f24a\") " pod="ai-tenant-e2e-shared-a-94a996/rate-test-model-94a996-kserve-84f5bc78b7-thzvn"
Jul 07 16:23:27.943569 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:27.943481    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72882acd-bf9d-437a-8d6d-178506f8f24a-tls-certs\") pod \"rate-test-model-94a996-kserve-84f5bc78b7-thzvn\" (UID: \"72882acd-bf9d-437a-8d6d-178506f8f24a\") " pod="ai-tenant-e2e-shared-a-94a996/rate-test-model-94a996-kserve-84f5bc78b7-thzvn"
Jul 07 16:23:27.943569 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:27.943509    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72882acd-bf9d-437a-8d6d-178506f8f24a-dshm\") pod \"rate-test-model-94a996-kserve-84f5bc78b7-thzvn\" (UID: \"72882acd-bf9d-437a-8d6d-178506f8f24a\") " pod="ai-tenant-e2e-shared-a-94a996/rate-test-model-94a996-kserve-84f5bc78b7-thzvn"
Jul 07 16:23:27.943723 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:27.943587    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72882acd-bf9d-437a-8d6d-178506f8f24a-model-cache\") pod \"rate-test-model-94a996-kserve-84f5bc78b7-thzvn\" (UID: \"72882acd-bf9d-437a-8d6d-178506f8f24a\") " pod="ai-tenant-e2e-shared-a-94a996/rate-test-model-94a996-kserve-84f5bc78b7-thzvn"
Jul 07 16:23:27.943723 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:27.943677    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72882acd-bf9d-437a-8d6d-178506f8f24a-home\") pod \"rate-test-model-94a996-kserve-84f5bc78b7-thzvn\" (UID: \"72882acd-bf9d-437a-8d6d-178506f8f24a\") " pod="ai-tenant-e2e-shared-a-94a996/rate-test-model-94a996-kserve-84f5bc78b7-thzvn"
Jul 07 16:23:27.943832 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:27.943772    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72882acd-bf9d-437a-8d6d-178506f8f24a-tmp-dir\") pod \"rate-test-model-94a996-kserve-84f5bc78b7-thzvn\" (UID: \"72882acd-bf9d-437a-8d6d-178506f8f24a\") " pod="ai-tenant-e2e-shared-a-94a996/rate-test-model-94a996-kserve-84f5bc78b7-thzvn"
Jul 07 16:23:27.943976 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:27.943958    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72882acd-bf9d-437a-8d6d-178506f8f24a-model-cache\") pod \"rate-test-model-94a996-kserve-84f5bc78b7-thzvn\" (UID: \"72882acd-bf9d-437a-8d6d-178506f8f24a\") " pod="ai-tenant-e2e-shared-a-94a996/rate-test-model-94a996-kserve-84f5bc78b7-thzvn"
Jul 07 16:23:27.945819 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:27.945796    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72882acd-bf9d-437a-8d6d-178506f8f24a-dshm\") pod \"rate-test-model-94a996-kserve-84f5bc78b7-thzvn\" (UID: \"72882acd-bf9d-437a-8d6d-178506f8f24a\") " pod="ai-tenant-e2e-shared-a-94a996/rate-test-model-94a996-kserve-84f5bc78b7-thzvn"
Jul 07 16:23:27.946064 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:27.946047    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72882acd-bf9d-437a-8d6d-178506f8f24a-tls-certs\") pod \"rate-test-model-94a996-kserve-84f5bc78b7-thzvn\" (UID: \"72882acd-bf9d-437a-8d6d-178506f8f24a\") " pod="ai-tenant-e2e-shared-a-94a996/rate-test-model-94a996-kserve-84f5bc78b7-thzvn"
Jul 07 16:23:27.951709 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:27.951682    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwp9b\" (UniqueName: \"kubernetes.io/projected/72882acd-bf9d-437a-8d6d-178506f8f24a-kube-api-access-pwp9b\") pod \"rate-test-model-94a996-kserve-84f5bc78b7-thzvn\" (UID: \"72882acd-bf9d-437a-8d6d-178506f8f24a\") " pod="ai-tenant-e2e-shared-a-94a996/rate-test-model-94a996-kserve-84f5bc78b7-thzvn"
Jul 07 16:23:28.055500 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:28.055396    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-94a996/rate-test-model-94a996-kserve-84f5bc78b7-thzvn"
Jul 07 16:23:28.198901 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:28.198850    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-94a996/rate-test-model-94a996-kserve-84f5bc78b7-thzvn"]
Jul 07 16:23:28.202841 ip-10-0-138-101 kubenswrapper[2566]: W0707 16:23:28.202811    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod72882acd_bf9d_437a_8d6d_178506f8f24a.slice/crio-201ceaaba31ff344d0eae1bbad84d12187a482a96987f24463fff9b48a7212da WatchSource:0}: Error finding container 201ceaaba31ff344d0eae1bbad84d12187a482a96987f24463fff9b48a7212da: Status 404 returned error can't find the container with id 201ceaaba31ff344d0eae1bbad84d12187a482a96987f24463fff9b48a7212da
Jul 07 16:23:29.093023 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:29.092991    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-94a996/rate-test-model-94a996-kserve-84f5bc78b7-thzvn" event={"ID":"72882acd-bf9d-437a-8d6d-178506f8f24a","Type":"ContainerStarted","Data":"1c6e92fe05d978e89c6e680ee8cec7dbd853cc8be62c964ae55fee36894a6b72"}
Jul 07 16:23:29.093023 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:29.093031    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-94a996/rate-test-model-94a996-kserve-84f5bc78b7-thzvn" event={"ID":"72882acd-bf9d-437a-8d6d-178506f8f24a","Type":"ContainerStarted","Data":"201ceaaba31ff344d0eae1bbad84d12187a482a96987f24463fff9b48a7212da"}
Jul 07 16:23:29.115442 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:29.115387    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-94a996/rate-test-model-94a996-kserve-84f5bc78b7-thzvn" podStartSLOduration=2.115372216 podStartE2EDuration="2.115372216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:23:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:23:29.11461303 +0000 UTC m=+2363.203038701" watchObservedRunningTime="2026-07-07 16:23:29.115372216 +0000 UTC m=+2363.203797886"
Jul 07 16:23:31.744161 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:31.744116    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mk99s"]
Jul 07 16:23:38.056006 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:38.055949    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-94a996/rate-test-model-94a996-kserve-84f5bc78b7-thzvn"
Jul 07 16:23:38.058403 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:38.056019    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-94a996/rate-test-model-94a996-kserve-84f5bc78b7-thzvn"
Jul 07 16:23:38.068619 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:38.068596    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-94a996/rate-test-model-94a996-kserve-84f5bc78b7-thzvn"
Jul 07 16:23:38.147636 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:38.147607    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-94a996/rate-test-model-94a996-kserve-84f5bc78b7-thzvn"
Jul 07 16:23:52.649930 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:52.649860    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-1cea82/rate-test-model-1cea82-kserve-69f98f586c-mrqkg"]
Jul 07 16:23:52.653779 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:52.653758    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1cea82/rate-test-model-1cea82-kserve-69f98f586c-mrqkg"
Jul 07 16:23:52.657356 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:52.657333    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-1cea82\"/\"kube-root-ca.crt\""
Jul 07 16:23:52.657356 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:52.657348    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-1cea82\"/\"openshift-service-ca.crt\""
Jul 07 16:23:52.657515 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:52.657408    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-1cea82\"/\"default-dockercfg-q6lbs\""
Jul 07 16:23:52.657515 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:52.657500    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-1cea82\"/\"rate-test-model-1cea82-kserve-self-signed-certs\""
Jul 07 16:23:52.663688 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:52.663667    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-1cea82/rate-test-model-1cea82-kserve-69f98f586c-mrqkg"]
Jul 07 16:23:52.755647 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:52.755609    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a61b604-4f78-4d6e-bf5c-11325716c60c-tmp-dir\") pod \"rate-test-model-1cea82-kserve-69f98f586c-mrqkg\" (UID: \"0a61b604-4f78-4d6e-bf5c-11325716c60c\") " pod="ai-tenant-e2e-shared-b-1cea82/rate-test-model-1cea82-kserve-69f98f586c-mrqkg"
Jul 07 16:23:52.755839 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:52.755754    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a61b604-4f78-4d6e-bf5c-11325716c60c-dshm\") pod \"rate-test-model-1cea82-kserve-69f98f586c-mrqkg\" (UID: \"0a61b604-4f78-4d6e-bf5c-11325716c60c\") " pod="ai-tenant-e2e-shared-b-1cea82/rate-test-model-1cea82-kserve-69f98f586c-mrqkg"
Jul 07 16:23:52.755839 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:52.755802    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a61b604-4f78-4d6e-bf5c-11325716c60c-tls-certs\") pod \"rate-test-model-1cea82-kserve-69f98f586c-mrqkg\" (UID: \"0a61b604-4f78-4d6e-bf5c-11325716c60c\") " pod="ai-tenant-e2e-shared-b-1cea82/rate-test-model-1cea82-kserve-69f98f586c-mrqkg"
Jul 07 16:23:52.755839 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:52.755831    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a61b604-4f78-4d6e-bf5c-11325716c60c-home\") pod \"rate-test-model-1cea82-kserve-69f98f586c-mrqkg\" (UID: \"0a61b604-4f78-4d6e-bf5c-11325716c60c\") " pod="ai-tenant-e2e-shared-b-1cea82/rate-test-model-1cea82-kserve-69f98f586c-mrqkg"
Jul 07 16:23:52.756026 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:52.755901    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9rkn\" (UniqueName: \"kubernetes.io/projected/0a61b604-4f78-4d6e-bf5c-11325716c60c-kube-api-access-r9rkn\") pod \"rate-test-model-1cea82-kserve-69f98f586c-mrqkg\" (UID: \"0a61b604-4f78-4d6e-bf5c-11325716c60c\") " pod="ai-tenant-e2e-shared-b-1cea82/rate-test-model-1cea82-kserve-69f98f586c-mrqkg"
Jul 07 16:23:52.756026 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:52.755933    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a61b604-4f78-4d6e-bf5c-11325716c60c-model-cache\") pod \"rate-test-model-1cea82-kserve-69f98f586c-mrqkg\" (UID: \"0a61b604-4f78-4d6e-bf5c-11325716c60c\") " pod="ai-tenant-e2e-shared-b-1cea82/rate-test-model-1cea82-kserve-69f98f586c-mrqkg"
Jul 07 16:23:52.856474 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:52.856439    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a61b604-4f78-4d6e-bf5c-11325716c60c-dshm\") pod \"rate-test-model-1cea82-kserve-69f98f586c-mrqkg\" (UID: \"0a61b604-4f78-4d6e-bf5c-11325716c60c\") " pod="ai-tenant-e2e-shared-b-1cea82/rate-test-model-1cea82-kserve-69f98f586c-mrqkg"
Jul 07 16:23:52.856685 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:52.856492    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a61b604-4f78-4d6e-bf5c-11325716c60c-tls-certs\") pod \"rate-test-model-1cea82-kserve-69f98f586c-mrqkg\" (UID: \"0a61b604-4f78-4d6e-bf5c-11325716c60c\") " pod="ai-tenant-e2e-shared-b-1cea82/rate-test-model-1cea82-kserve-69f98f586c-mrqkg"
Jul 07 16:23:52.856685 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:52.856522    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a61b604-4f78-4d6e-bf5c-11325716c60c-home\") pod \"rate-test-model-1cea82-kserve-69f98f586c-mrqkg\" (UID: \"0a61b604-4f78-4d6e-bf5c-11325716c60c\") " pod="ai-tenant-e2e-shared-b-1cea82/rate-test-model-1cea82-kserve-69f98f586c-mrqkg"
Jul 07 16:23:52.856685 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:52.856557    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9rkn\" (UniqueName: \"kubernetes.io/projected/0a61b604-4f78-4d6e-bf5c-11325716c60c-kube-api-access-r9rkn\") pod \"rate-test-model-1cea82-kserve-69f98f586c-mrqkg\" (UID: \"0a61b604-4f78-4d6e-bf5c-11325716c60c\") " pod="ai-tenant-e2e-shared-b-1cea82/rate-test-model-1cea82-kserve-69f98f586c-mrqkg"
Jul 07 16:23:52.856685 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:52.856579    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a61b604-4f78-4d6e-bf5c-11325716c60c-model-cache\") pod \"rate-test-model-1cea82-kserve-69f98f586c-mrqkg\" (UID: \"0a61b604-4f78-4d6e-bf5c-11325716c60c\") " pod="ai-tenant-e2e-shared-b-1cea82/rate-test-model-1cea82-kserve-69f98f586c-mrqkg"
Jul 07 16:23:52.856685 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:52.856641    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a61b604-4f78-4d6e-bf5c-11325716c60c-tmp-dir\") pod \"rate-test-model-1cea82-kserve-69f98f586c-mrqkg\" (UID: \"0a61b604-4f78-4d6e-bf5c-11325716c60c\") " pod="ai-tenant-e2e-shared-b-1cea82/rate-test-model-1cea82-kserve-69f98f586c-mrqkg"
Jul 07 16:23:52.857020 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:52.856990    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a61b604-4f78-4d6e-bf5c-11325716c60c-home\") pod \"rate-test-model-1cea82-kserve-69f98f586c-mrqkg\" (UID: \"0a61b604-4f78-4d6e-bf5c-11325716c60c\") " pod="ai-tenant-e2e-shared-b-1cea82/rate-test-model-1cea82-kserve-69f98f586c-mrqkg"
Jul 07 16:23:52.857143 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:52.857053    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a61b604-4f78-4d6e-bf5c-11325716c60c-tmp-dir\") pod \"rate-test-model-1cea82-kserve-69f98f586c-mrqkg\" (UID: \"0a61b604-4f78-4d6e-bf5c-11325716c60c\") " pod="ai-tenant-e2e-shared-b-1cea82/rate-test-model-1cea82-kserve-69f98f586c-mrqkg"
Jul 07 16:23:52.857223 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:52.857206    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a61b604-4f78-4d6e-bf5c-11325716c60c-model-cache\") pod \"rate-test-model-1cea82-kserve-69f98f586c-mrqkg\" (UID: \"0a61b604-4f78-4d6e-bf5c-11325716c60c\") " pod="ai-tenant-e2e-shared-b-1cea82/rate-test-model-1cea82-kserve-69f98f586c-mrqkg"
Jul 07 16:23:52.859405 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:52.859382    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a61b604-4f78-4d6e-bf5c-11325716c60c-dshm\") pod \"rate-test-model-1cea82-kserve-69f98f586c-mrqkg\" (UID: \"0a61b604-4f78-4d6e-bf5c-11325716c60c\") " pod="ai-tenant-e2e-shared-b-1cea82/rate-test-model-1cea82-kserve-69f98f586c-mrqkg"
Jul 07 16:23:52.859507 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:52.859439    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a61b604-4f78-4d6e-bf5c-11325716c60c-tls-certs\") pod \"rate-test-model-1cea82-kserve-69f98f586c-mrqkg\" (UID: \"0a61b604-4f78-4d6e-bf5c-11325716c60c\") " pod="ai-tenant-e2e-shared-b-1cea82/rate-test-model-1cea82-kserve-69f98f586c-mrqkg"
Jul 07 16:23:52.866197 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:52.866172    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9rkn\" (UniqueName: \"kubernetes.io/projected/0a61b604-4f78-4d6e-bf5c-11325716c60c-kube-api-access-r9rkn\") pod \"rate-test-model-1cea82-kserve-69f98f586c-mrqkg\" (UID: \"0a61b604-4f78-4d6e-bf5c-11325716c60c\") " pod="ai-tenant-e2e-shared-b-1cea82/rate-test-model-1cea82-kserve-69f98f586c-mrqkg"
Jul 07 16:23:52.965133 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:52.965093    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1cea82/rate-test-model-1cea82-kserve-69f98f586c-mrqkg"
Jul 07 16:23:53.108835 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:53.108804    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-1cea82/rate-test-model-1cea82-kserve-69f98f586c-mrqkg"]
Jul 07 16:23:53.111322 ip-10-0-138-101 kubenswrapper[2566]: W0707 16:23:53.111285    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0a61b604_4f78_4d6e_bf5c_11325716c60c.slice/crio-db9b4dee9ea98f419b7f0bb9b82baa3b9fdbb48e9850b089c5077f9e80ec37e2 WatchSource:0}: Error finding container db9b4dee9ea98f419b7f0bb9b82baa3b9fdbb48e9850b089c5077f9e80ec37e2: Status 404 returned error can't find the container with id db9b4dee9ea98f419b7f0bb9b82baa3b9fdbb48e9850b089c5077f9e80ec37e2
Jul 07 16:23:53.212411 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:53.212371    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1cea82/rate-test-model-1cea82-kserve-69f98f586c-mrqkg" event={"ID":"0a61b604-4f78-4d6e-bf5c-11325716c60c","Type":"ContainerStarted","Data":"b7c382fd9fbfc7a5ea1ae130eb104fce66c186d2321b482b685bba52e6fd8912"}
Jul 07 16:23:53.212411 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:53.212421    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1cea82/rate-test-model-1cea82-kserve-69f98f586c-mrqkg" event={"ID":"0a61b604-4f78-4d6e-bf5c-11325716c60c","Type":"ContainerStarted","Data":"db9b4dee9ea98f419b7f0bb9b82baa3b9fdbb48e9850b089c5077f9e80ec37e2"}
Jul 07 16:23:53.235430 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:53.235327    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-1cea82/rate-test-model-1cea82-kserve-69f98f586c-mrqkg" podStartSLOduration=1.235304246 podStartE2EDuration="1.235304246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:23:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:23:53.234306387 +0000 UTC m=+2387.322732058" watchObservedRunningTime="2026-07-07 16:23:53.235304246 +0000 UTC m=+2387.323729916"
Jul 07 16:23:56.934299 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:23:56.934264    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mk99s"]
Jul 07 16:24:02.965260 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:24:02.965218    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-1cea82/rate-test-model-1cea82-kserve-69f98f586c-mrqkg"
Jul 07 16:24:02.965668 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:24:02.965283    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-1cea82/rate-test-model-1cea82-kserve-69f98f586c-mrqkg"
Jul 07 16:24:02.981524 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:24:02.981493    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-1cea82/rate-test-model-1cea82-kserve-69f98f586c-mrqkg"
Jul 07 16:24:03.269090 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:24:03.268989    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-1cea82/rate-test-model-1cea82-kserve-69f98f586c-mrqkg"
Jul 07 16:24:16.037168 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:24:16.037133    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mk99s"]
Jul 07 16:24:30.632353 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:24:30.632267    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mk99s"]
Jul 07 16:24:37.014544 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:24:37.014506    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-94a996/rate-test-model-94a996-kserve-84f5bc78b7-thzvn"]
Jul 07 16:24:38.012860 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:24:38.012823    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-1cea82/rate-test-model-1cea82-kserve-69f98f586c-mrqkg"]
Jul 07 16:24:43.840981 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:24:43.840943    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mk99s"]
Jul 07 16:24:52.037989 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:24:52.037927    2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-94a996/rate-test-model-94a996-kserve-84f5bc78b7-thzvn" podUID="72882acd-bf9d-437a-8d6d-178506f8f24a" containerName="main" containerID="cri-o://1c6e92fe05d978e89c6e680ee8cec7dbd853cc8be62c964ae55fee36894a6b72" gracePeriod=45
Jul 07 16:24:52.309655 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:24:52.309625    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-94a996/rate-test-model-94a996-kserve-84f5bc78b7-thzvn"
Jul 07 16:24:52.400294 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:24:52.400260    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pwp9b\" (UniqueName: \"kubernetes.io/projected/72882acd-bf9d-437a-8d6d-178506f8f24a-kube-api-access-pwp9b\") pod \"72882acd-bf9d-437a-8d6d-178506f8f24a\" (UID: \"72882acd-bf9d-437a-8d6d-178506f8f24a\") "
Jul 07 16:24:52.400294 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:24:52.400300    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72882acd-bf9d-437a-8d6d-178506f8f24a-model-cache\") pod \"72882acd-bf9d-437a-8d6d-178506f8f24a\" (UID: \"72882acd-bf9d-437a-8d6d-178506f8f24a\") "
Jul 07 16:24:52.400556 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:24:52.400325    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72882acd-bf9d-437a-8d6d-178506f8f24a-tmp-dir\") pod \"72882acd-bf9d-437a-8d6d-178506f8f24a\" (UID: \"72882acd-bf9d-437a-8d6d-178506f8f24a\") "
Jul 07 16:24:52.400556 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:24:52.400358    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72882acd-bf9d-437a-8d6d-178506f8f24a-dshm\") pod \"72882acd-bf9d-437a-8d6d-178506f8f24a\" (UID: \"72882acd-bf9d-437a-8d6d-178506f8f24a\") "
Jul 07 16:24:52.400556 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:24:52.400384    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72882acd-bf9d-437a-8d6d-178506f8f24a-tls-certs\") pod \"72882acd-bf9d-437a-8d6d-178506f8f24a\" (UID: \"72882acd-bf9d-437a-8d6d-178506f8f24a\") "
Jul 07 16:24:52.400556 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:24:52.400414    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72882acd-bf9d-437a-8d6d-178506f8f24a-home\") pod \"72882acd-bf9d-437a-8d6d-178506f8f24a\" (UID: \"72882acd-bf9d-437a-8d6d-178506f8f24a\") "
Jul 07 16:24:52.400767 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:24:52.400563    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72882acd-bf9d-437a-8d6d-178506f8f24a-model-cache" (OuterVolumeSpecName: "model-cache") pod "72882acd-bf9d-437a-8d6d-178506f8f24a" (UID: "72882acd-bf9d-437a-8d6d-178506f8f24a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 16:24:52.400767 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:24:52.400584    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72882acd-bf9d-437a-8d6d-178506f8f24a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "72882acd-bf9d-437a-8d6d-178506f8f24a" (UID: "72882acd-bf9d-437a-8d6d-178506f8f24a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 16:24:52.400767 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:24:52.400759    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72882acd-bf9d-437a-8d6d-178506f8f24a-model-cache\") on node \"ip-10-0-138-101.ec2.internal\" DevicePath \"\""
Jul 07 16:24:52.400947 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:24:52.400779    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72882acd-bf9d-437a-8d6d-178506f8f24a-tmp-dir\") on node \"ip-10-0-138-101.ec2.internal\" DevicePath \"\""
Jul 07 16:24:52.400947 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:24:52.400820    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72882acd-bf9d-437a-8d6d-178506f8f24a-home" (OuterVolumeSpecName: "home") pod "72882acd-bf9d-437a-8d6d-178506f8f24a" (UID: "72882acd-bf9d-437a-8d6d-178506f8f24a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 16:24:52.402570 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:24:52.402539    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72882acd-bf9d-437a-8d6d-178506f8f24a-dshm" (OuterVolumeSpecName: "dshm") pod "72882acd-bf9d-437a-8d6d-178506f8f24a" (UID: "72882acd-bf9d-437a-8d6d-178506f8f24a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 16:24:52.402693 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:24:52.402622    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72882acd-bf9d-437a-8d6d-178506f8f24a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "72882acd-bf9d-437a-8d6d-178506f8f24a" (UID: "72882acd-bf9d-437a-8d6d-178506f8f24a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 16:24:52.402693 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:24:52.402684    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72882acd-bf9d-437a-8d6d-178506f8f24a-kube-api-access-pwp9b" (OuterVolumeSpecName: "kube-api-access-pwp9b") pod "72882acd-bf9d-437a-8d6d-178506f8f24a" (UID: "72882acd-bf9d-437a-8d6d-178506f8f24a"). InnerVolumeSpecName "kube-api-access-pwp9b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 16:24:52.487600 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:24:52.487573    2566 generic.go:356] "Generic (PLEG): container finished" podID="72882acd-bf9d-437a-8d6d-178506f8f24a" containerID="1c6e92fe05d978e89c6e680ee8cec7dbd853cc8be62c964ae55fee36894a6b72" exitCode=0
Jul 07 16:24:52.487760 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:24:52.487617    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-94a996/rate-test-model-94a996-kserve-84f5bc78b7-thzvn"
Jul 07 16:24:52.487760 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:24:52.487657    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-94a996/rate-test-model-94a996-kserve-84f5bc78b7-thzvn" event={"ID":"72882acd-bf9d-437a-8d6d-178506f8f24a","Type":"ContainerDied","Data":"1c6e92fe05d978e89c6e680ee8cec7dbd853cc8be62c964ae55fee36894a6b72"}
Jul 07 16:24:52.487760 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:24:52.487706    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-94a996/rate-test-model-94a996-kserve-84f5bc78b7-thzvn" event={"ID":"72882acd-bf9d-437a-8d6d-178506f8f24a","Type":"ContainerDied","Data":"201ceaaba31ff344d0eae1bbad84d12187a482a96987f24463fff9b48a7212da"}
Jul 07 16:24:52.487760 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:24:52.487728    2566 scope.go:117] "RemoveContainer" containerID="1c6e92fe05d978e89c6e680ee8cec7dbd853cc8be62c964ae55fee36894a6b72"
Jul 07 16:24:52.496385 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:24:52.496365    2566 scope.go:117] "RemoveContainer" containerID="1c6e92fe05d978e89c6e680ee8cec7dbd853cc8be62c964ae55fee36894a6b72"
Jul 07 16:24:52.496649 ip-10-0-138-101 kubenswrapper[2566]: E0707 16:24:52.496630    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c6e92fe05d978e89c6e680ee8cec7dbd853cc8be62c964ae55fee36894a6b72\": container with ID starting with 1c6e92fe05d978e89c6e680ee8cec7dbd853cc8be62c964ae55fee36894a6b72 not found: ID does not exist" containerID="1c6e92fe05d978e89c6e680ee8cec7dbd853cc8be62c964ae55fee36894a6b72"
Jul 07 16:24:52.496702 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:24:52.496657    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c6e92fe05d978e89c6e680ee8cec7dbd853cc8be62c964ae55fee36894a6b72"} err="failed to get container status \"1c6e92fe05d978e89c6e680ee8cec7dbd853cc8be62c964ae55fee36894a6b72\": rpc error: code = NotFound desc = could not find container \"1c6e92fe05d978e89c6e680ee8cec7dbd853cc8be62c964ae55fee36894a6b72\": container with ID starting with 1c6e92fe05d978e89c6e680ee8cec7dbd853cc8be62c964ae55fee36894a6b72 not found: ID does not exist"
Jul 07 16:24:52.501210 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:24:52.501187    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72882acd-bf9d-437a-8d6d-178506f8f24a-dshm\") on node \"ip-10-0-138-101.ec2.internal\" DevicePath \"\""
Jul 07 16:24:52.501210 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:24:52.501209    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72882acd-bf9d-437a-8d6d-178506f8f24a-tls-certs\") on node \"ip-10-0-138-101.ec2.internal\" DevicePath \"\""
Jul 07 16:24:52.501367 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:24:52.501218    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72882acd-bf9d-437a-8d6d-178506f8f24a-home\") on node \"ip-10-0-138-101.ec2.internal\" DevicePath \"\""
Jul 07 16:24:52.501367 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:24:52.501227    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwp9b\" (UniqueName: \"kubernetes.io/projected/72882acd-bf9d-437a-8d6d-178506f8f24a-kube-api-access-pwp9b\") on node \"ip-10-0-138-101.ec2.internal\" DevicePath \"\""
Jul 07 16:24:52.506949 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:24:52.506929    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-94a996/rate-test-model-94a996-kserve-84f5bc78b7-thzvn"]
Jul 07 16:24:52.509540 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:24:52.509518    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-94a996/rate-test-model-94a996-kserve-84f5bc78b7-thzvn"]
Jul 07 16:24:53.033057 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:24:53.033010    2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-1cea82/rate-test-model-1cea82-kserve-69f98f586c-mrqkg" podUID="0a61b604-4f78-4d6e-bf5c-11325716c60c" containerName="main" containerID="cri-o://b7c382fd9fbfc7a5ea1ae130eb104fce66c186d2321b482b685bba52e6fd8912" gracePeriod=45
Jul 07 16:24:53.064688 ip-10-0-138-101 kubenswrapper[2566]: E0707 16:24:53.064654    2566 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0a61b604_4f78_4d6e_bf5c_11325716c60c.slice/crio-b7c382fd9fbfc7a5ea1ae130eb104fce66c186d2321b482b685bba52e6fd8912.scope\": RecentStats: unable to find data in memory cache]"
Jul 07 16:24:53.291904 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:24:53.291827    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1cea82/rate-test-model-1cea82-kserve-69f98f586c-mrqkg"
Jul 07 16:24:53.306897 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:24:53.306863    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a61b604-4f78-4d6e-bf5c-11325716c60c-model-cache\") pod \"0a61b604-4f78-4d6e-bf5c-11325716c60c\" (UID: \"0a61b604-4f78-4d6e-bf5c-11325716c60c\") "
Jul 07 16:24:53.307050 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:24:53.306954    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a61b604-4f78-4d6e-bf5c-11325716c60c-tmp-dir\") pod \"0a61b604-4f78-4d6e-bf5c-11325716c60c\" (UID: \"0a61b604-4f78-4d6e-bf5c-11325716c60c\") "
Jul 07 16:24:53.307050 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:24:53.306977    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r9rkn\" (UniqueName: \"kubernetes.io/projected/0a61b604-4f78-4d6e-bf5c-11325716c60c-kube-api-access-r9rkn\") pod \"0a61b604-4f78-4d6e-bf5c-11325716c60c\" (UID: \"0a61b604-4f78-4d6e-bf5c-11325716c60c\") "
Jul 07 16:24:53.307050 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:24:53.307006    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a61b604-4f78-4d6e-bf5c-11325716c60c-dshm\") pod \"0a61b604-4f78-4d6e-bf5c-11325716c60c\" (UID: \"0a61b604-4f78-4d6e-bf5c-11325716c60c\") "
Jul 07 16:24:53.307050 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:24:53.307043    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a61b604-4f78-4d6e-bf5c-11325716c60c-tls-certs\") pod \"0a61b604-4f78-4d6e-bf5c-11325716c60c\" (UID: \"0a61b604-4f78-4d6e-bf5c-11325716c60c\") "
Jul 07 16:24:53.307262 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:24:53.307104    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a61b604-4f78-4d6e-bf5c-11325716c60c-home\") pod \"0a61b604-4f78-4d6e-bf5c-11325716c60c\" (UID: \"0a61b604-4f78-4d6e-bf5c-11325716c60c\") "
Jul 07 16:24:53.307262 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:24:53.307189    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a61b604-4f78-4d6e-bf5c-11325716c60c-model-cache" (OuterVolumeSpecName: "model-cache") pod "0a61b604-4f78-4d6e-bf5c-11325716c60c" (UID: "0a61b604-4f78-4d6e-bf5c-11325716c60c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 16:24:53.307262 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:24:53.307250    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a61b604-4f78-4d6e-bf5c-11325716c60c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0a61b604-4f78-4d6e-bf5c-11325716c60c" (UID: "0a61b604-4f78-4d6e-bf5c-11325716c60c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 16:24:53.307409 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:24:53.307390    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a61b604-4f78-4d6e-bf5c-11325716c60c-home" (OuterVolumeSpecName: "home") pod "0a61b604-4f78-4d6e-bf5c-11325716c60c" (UID: "0a61b604-4f78-4d6e-bf5c-11325716c60c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 16:24:53.307482 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:24:53.307461    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a61b604-4f78-4d6e-bf5c-11325716c60c-model-cache\") on node \"ip-10-0-138-101.ec2.internal\" DevicePath \"\""
Jul 07 16:24:53.307482 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:24:53.307482    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a61b604-4f78-4d6e-bf5c-11325716c60c-tmp-dir\") on node \"ip-10-0-138-101.ec2.internal\" DevicePath \"\""
Jul 07 16:24:53.309202 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:24:53.309181    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a61b604-4f78-4d6e-bf5c-11325716c60c-dshm" (OuterVolumeSpecName: "dshm") pod "0a61b604-4f78-4d6e-bf5c-11325716c60c" (UID: "0a61b604-4f78-4d6e-bf5c-11325716c60c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 16:24:53.309282 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:24:53.309218    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a61b604-4f78-4d6e-bf5c-11325716c60c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0a61b604-4f78-4d6e-bf5c-11325716c60c" (UID: "0a61b604-4f78-4d6e-bf5c-11325716c60c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 16:24:53.309322 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:24:53.309290    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a61b604-4f78-4d6e-bf5c-11325716c60c-kube-api-access-r9rkn" (OuterVolumeSpecName: "kube-api-access-r9rkn") pod "0a61b604-4f78-4d6e-bf5c-11325716c60c" (UID: "0a61b604-4f78-4d6e-bf5c-11325716c60c"). InnerVolumeSpecName "kube-api-access-r9rkn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 16:24:53.408110 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:24:53.408076    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9rkn\" (UniqueName: \"kubernetes.io/projected/0a61b604-4f78-4d6e-bf5c-11325716c60c-kube-api-access-r9rkn\") on node \"ip-10-0-138-101.ec2.internal\" DevicePath \"\""
Jul 07 16:24:53.408110 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:24:53.408107    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a61b604-4f78-4d6e-bf5c-11325716c60c-dshm\") on node \"ip-10-0-138-101.ec2.internal\" DevicePath \"\""
Jul 07 16:24:53.408324 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:24:53.408119    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a61b604-4f78-4d6e-bf5c-11325716c60c-tls-certs\") on node \"ip-10-0-138-101.ec2.internal\" DevicePath \"\""
Jul 07 16:24:53.408324 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:24:53.408128    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a61b604-4f78-4d6e-bf5c-11325716c60c-home\") on node \"ip-10-0-138-101.ec2.internal\" DevicePath \"\""
Jul 07 16:24:53.494222 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:24:53.494188    2566 generic.go:356] "Generic (PLEG): container finished" podID="0a61b604-4f78-4d6e-bf5c-11325716c60c" containerID="b7c382fd9fbfc7a5ea1ae130eb104fce66c186d2321b482b685bba52e6fd8912" exitCode=0
Jul 07 16:24:53.494409 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:24:53.494229    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1cea82/rate-test-model-1cea82-kserve-69f98f586c-mrqkg"
Jul 07 16:24:53.494409 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:24:53.494263    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1cea82/rate-test-model-1cea82-kserve-69f98f586c-mrqkg" event={"ID":"0a61b604-4f78-4d6e-bf5c-11325716c60c","Type":"ContainerDied","Data":"b7c382fd9fbfc7a5ea1ae130eb104fce66c186d2321b482b685bba52e6fd8912"}
Jul 07 16:24:53.494409 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:24:53.494307    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1cea82/rate-test-model-1cea82-kserve-69f98f586c-mrqkg" event={"ID":"0a61b604-4f78-4d6e-bf5c-11325716c60c","Type":"ContainerDied","Data":"db9b4dee9ea98f419b7f0bb9b82baa3b9fdbb48e9850b089c5077f9e80ec37e2"}
Jul 07 16:24:53.494409 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:24:53.494326    2566 scope.go:117] "RemoveContainer" containerID="b7c382fd9fbfc7a5ea1ae130eb104fce66c186d2321b482b685bba52e6fd8912"
Jul 07 16:24:53.503511 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:24:53.503404    2566 scope.go:117] "RemoveContainer" containerID="b7c382fd9fbfc7a5ea1ae130eb104fce66c186d2321b482b685bba52e6fd8912"
Jul 07 16:24:53.503861 ip-10-0-138-101 kubenswrapper[2566]: E0707 16:24:53.503834    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7c382fd9fbfc7a5ea1ae130eb104fce66c186d2321b482b685bba52e6fd8912\": container with ID starting with b7c382fd9fbfc7a5ea1ae130eb104fce66c186d2321b482b685bba52e6fd8912 not found: ID does not exist" containerID="b7c382fd9fbfc7a5ea1ae130eb104fce66c186d2321b482b685bba52e6fd8912"
Jul 07 16:24:53.504058 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:24:53.503871    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7c382fd9fbfc7a5ea1ae130eb104fce66c186d2321b482b685bba52e6fd8912"} err="failed to get container status \"b7c382fd9fbfc7a5ea1ae130eb104fce66c186d2321b482b685bba52e6fd8912\": rpc error: code = NotFound desc = could not find container \"b7c382fd9fbfc7a5ea1ae130eb104fce66c186d2321b482b685bba52e6fd8912\": container with ID starting with b7c382fd9fbfc7a5ea1ae130eb104fce66c186d2321b482b685bba52e6fd8912 not found: ID does not exist"
Jul 07 16:24:53.518483 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:24:53.518458    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-1cea82/rate-test-model-1cea82-kserve-69f98f586c-mrqkg"]
Jul 07 16:24:53.522859 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:24:53.522834    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-1cea82/rate-test-model-1cea82-kserve-69f98f586c-mrqkg"]
Jul 07 16:24:54.474454 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:24:54.474420    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a61b604-4f78-4d6e-bf5c-11325716c60c" path="/var/lib/kubelet/pods/0a61b604-4f78-4d6e-bf5c-11325716c60c/volumes"
Jul 07 16:24:54.474849 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:24:54.474814    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72882acd-bf9d-437a-8d6d-178506f8f24a" path="/var/lib/kubelet/pods/72882acd-bf9d-437a-8d6d-178506f8f24a/volumes"
Jul 07 16:24:58.258811 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:24:58.258769    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-1cea82/rate-test-model-1cea82-kserve-69f98f586c-mrqkg" podUID="0a61b604-4f78-4d6e-bf5c-11325716c60c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Jul 07 16:25:24.034601 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:24.034562    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mk99s"]
Jul 07 16:25:27.468630 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:27.468592    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-1cea82-openshift-default-6ff8898c79-7mshg"]
Jul 07 16:25:27.469248 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:27.468867    2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-1cea82-openshift-default-6ff8898c79-7mshg" podUID="f28c34bc-0591-4290-9d69-3f8a1cb95213" containerName="istio-proxy" containerID="cri-o://65c12247db59c5aee938b9789d4d297d13d380fa11448e60593099ff1e5e7680" gracePeriod=30
Jul 07 16:25:28.140392 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:28.140360    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-56b56968cc-gxml2_fde2c89a-f756-424d-9630-8993c5eee62c/authorino/0.log"
Jul 07 16:25:32.678205 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:32.678174    2566 generic.go:356] "Generic (PLEG): container finished" podID="f28c34bc-0591-4290-9d69-3f8a1cb95213" containerID="65c12247db59c5aee938b9789d4d297d13d380fa11448e60593099ff1e5e7680" exitCode=0
Jul 07 16:25:32.678608 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:32.678212    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-1cea82-openshift-default-6ff8898c79-7mshg" event={"ID":"f28c34bc-0591-4290-9d69-3f8a1cb95213","Type":"ContainerDied","Data":"65c12247db59c5aee938b9789d4d297d13d380fa11448e60593099ff1e5e7680"}
Jul 07 16:25:32.678608 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:32.678259    2566 scope.go:117] "RemoveContainer" containerID="e4270e88f42b2a38a291730bb377dc4a3e998cc54e8d09eb53a9f824295abdd8"
Jul 07 16:25:32.699533 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:32.699507    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-28lvw_cdc471c4-bece-4f05-9a9c-cf110a5eabb6/manager/0.log"
Jul 07 16:25:32.732987 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:32.732871    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-1cea82-openshift-default-6ff8898c79-7mshg"
Jul 07 16:25:32.821727 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:32.821700    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-b69vl_73a96c42-cb3f-4acf-adf4-31c21f32186c/manager/0.log"
Jul 07 16:25:32.858872 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:32.858838    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f28c34bc-0591-4290-9d69-3f8a1cb95213-workload-certs\") pod \"f28c34bc-0591-4290-9d69-3f8a1cb95213\" (UID: \"f28c34bc-0591-4290-9d69-3f8a1cb95213\") "
Jul 07 16:25:32.859001 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:32.858910    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f28c34bc-0591-4290-9d69-3f8a1cb95213-istio-envoy\") pod \"f28c34bc-0591-4290-9d69-3f8a1cb95213\" (UID: \"f28c34bc-0591-4290-9d69-3f8a1cb95213\") "
Jul 07 16:25:32.859001 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:32.858931    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f28c34bc-0591-4290-9d69-3f8a1cb95213-credential-socket\") pod \"f28c34bc-0591-4290-9d69-3f8a1cb95213\" (UID: \"f28c34bc-0591-4290-9d69-3f8a1cb95213\") "
Jul 07 16:25:32.859001 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:32.858958    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gcd26\" (UniqueName: \"kubernetes.io/projected/f28c34bc-0591-4290-9d69-3f8a1cb95213-kube-api-access-gcd26\") pod \"f28c34bc-0591-4290-9d69-3f8a1cb95213\" (UID: \"f28c34bc-0591-4290-9d69-3f8a1cb95213\") "
Jul 07 16:25:32.859141 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:32.859005    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f28c34bc-0591-4290-9d69-3f8a1cb95213-istio-token\") pod \"f28c34bc-0591-4290-9d69-3f8a1cb95213\" (UID: \"f28c34bc-0591-4290-9d69-3f8a1cb95213\") "
Jul 07 16:25:32.859141 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:32.859036    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f28c34bc-0591-4290-9d69-3f8a1cb95213-workload-socket\") pod \"f28c34bc-0591-4290-9d69-3f8a1cb95213\" (UID: \"f28c34bc-0591-4290-9d69-3f8a1cb95213\") "
Jul 07 16:25:32.859141 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:32.859063    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f28c34bc-0591-4290-9d69-3f8a1cb95213-istio-data\") pod \"f28c34bc-0591-4290-9d69-3f8a1cb95213\" (UID: \"f28c34bc-0591-4290-9d69-3f8a1cb95213\") "
Jul 07 16:25:32.859141 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:32.859111    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f28c34bc-0591-4290-9d69-3f8a1cb95213-istio-podinfo\") pod \"f28c34bc-0591-4290-9d69-3f8a1cb95213\" (UID: \"f28c34bc-0591-4290-9d69-3f8a1cb95213\") "
Jul 07 16:25:32.859141 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:32.859127    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f28c34bc-0591-4290-9d69-3f8a1cb95213-istiod-ca-cert\") pod \"f28c34bc-0591-4290-9d69-3f8a1cb95213\" (UID: \"f28c34bc-0591-4290-9d69-3f8a1cb95213\") "
Jul 07 16:25:32.859461 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:32.859151    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f28c34bc-0591-4290-9d69-3f8a1cb95213-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f28c34bc-0591-4290-9d69-3f8a1cb95213" (UID: "f28c34bc-0591-4290-9d69-3f8a1cb95213"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 16:25:32.859461 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:32.859389    2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f28c34bc-0591-4290-9d69-3f8a1cb95213-workload-certs\") on node \"ip-10-0-138-101.ec2.internal\" DevicePath \"\""
Jul 07 16:25:32.859574 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:32.859473    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f28c34bc-0591-4290-9d69-3f8a1cb95213-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f28c34bc-0591-4290-9d69-3f8a1cb95213" (UID: "f28c34bc-0591-4290-9d69-3f8a1cb95213"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 16:25:32.859574 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:32.859544    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f28c34bc-0591-4290-9d69-3f8a1cb95213-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f28c34bc-0591-4290-9d69-3f8a1cb95213" (UID: "f28c34bc-0591-4290-9d69-3f8a1cb95213"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 16:25:32.859688 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:32.859659    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f28c34bc-0591-4290-9d69-3f8a1cb95213-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f28c34bc-0591-4290-9d69-3f8a1cb95213" (UID: "f28c34bc-0591-4290-9d69-3f8a1cb95213"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 16:25:32.859960 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:32.859934    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f28c34bc-0591-4290-9d69-3f8a1cb95213-istio-data" (OuterVolumeSpecName: "istio-data") pod "f28c34bc-0591-4290-9d69-3f8a1cb95213" (UID: "f28c34bc-0591-4290-9d69-3f8a1cb95213"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 16:25:32.861211 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:32.861189    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f28c34bc-0591-4290-9d69-3f8a1cb95213-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f28c34bc-0591-4290-9d69-3f8a1cb95213" (UID: "f28c34bc-0591-4290-9d69-3f8a1cb95213"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 16:25:32.861313 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:32.861257    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f28c34bc-0591-4290-9d69-3f8a1cb95213-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f28c34bc-0591-4290-9d69-3f8a1cb95213" (UID: "f28c34bc-0591-4290-9d69-3f8a1cb95213"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 07 16:25:32.861604 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:32.861571    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f28c34bc-0591-4290-9d69-3f8a1cb95213-istio-token" (OuterVolumeSpecName: "istio-token") pod "f28c34bc-0591-4290-9d69-3f8a1cb95213" (UID: "f28c34bc-0591-4290-9d69-3f8a1cb95213"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 16:25:32.861604 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:32.861571    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f28c34bc-0591-4290-9d69-3f8a1cb95213-kube-api-access-gcd26" (OuterVolumeSpecName: "kube-api-access-gcd26") pod "f28c34bc-0591-4290-9d69-3f8a1cb95213" (UID: "f28c34bc-0591-4290-9d69-3f8a1cb95213"). InnerVolumeSpecName "kube-api-access-gcd26". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 16:25:32.960955 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:32.960916    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcd26\" (UniqueName: \"kubernetes.io/projected/f28c34bc-0591-4290-9d69-3f8a1cb95213-kube-api-access-gcd26\") on node \"ip-10-0-138-101.ec2.internal\" DevicePath \"\""
Jul 07 16:25:32.960955 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:32.960956    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f28c34bc-0591-4290-9d69-3f8a1cb95213-istio-token\") on node \"ip-10-0-138-101.ec2.internal\" DevicePath \"\""
Jul 07 16:25:32.961197 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:32.960972    2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f28c34bc-0591-4290-9d69-3f8a1cb95213-workload-socket\") on node \"ip-10-0-138-101.ec2.internal\" DevicePath \"\""
Jul 07 16:25:32.961197 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:32.960986    2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f28c34bc-0591-4290-9d69-3f8a1cb95213-istio-data\") on node \"ip-10-0-138-101.ec2.internal\" DevicePath \"\""
Jul 07 16:25:32.961197 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:32.960998    2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f28c34bc-0591-4290-9d69-3f8a1cb95213-istio-podinfo\") on node \"ip-10-0-138-101.ec2.internal\" DevicePath \"\""
Jul 07 16:25:32.961197 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:32.961006    2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f28c34bc-0591-4290-9d69-3f8a1cb95213-istiod-ca-cert\") on node \"ip-10-0-138-101.ec2.internal\" DevicePath \"\""
Jul 07 16:25:32.961197 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:32.961015    2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f28c34bc-0591-4290-9d69-3f8a1cb95213-istio-envoy\") on node \"ip-10-0-138-101.ec2.internal\" DevicePath \"\""
Jul 07 16:25:32.961197 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:32.961023    2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f28c34bc-0591-4290-9d69-3f8a1cb95213-credential-socket\") on node \"ip-10-0-138-101.ec2.internal\" DevicePath \"\""
Jul 07 16:25:33.079757 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:33.079677    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-nkdqs_e5962e3e-69f6-42f4-b642-7bac15a37ede/server/0.log"
Jul 07 16:25:33.201080 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:33.201047    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-kt87x_d1ce20f8-4500-4af3-b2e9-d5c6933df650/manager/0.log"
Jul 07 16:25:33.551317 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:33.551286    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-56f57b78d4-tbrcq_7b3312b8-7ba2-4102-b36f-3b0f58178f4e/copy-manifests/0.log"
Jul 07 16:25:33.565465 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:33.565430    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-56f57b78d4-tbrcq_7b3312b8-7ba2-4102-b36f-3b0f58178f4e/manager/0.log"
Jul 07 16:25:33.684596 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:33.684564    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-1cea82-openshift-default-6ff8898c79-7mshg"
Jul 07 16:25:33.685107 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:33.684562    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-1cea82-openshift-default-6ff8898c79-7mshg" event={"ID":"f28c34bc-0591-4290-9d69-3f8a1cb95213","Type":"ContainerDied","Data":"22a4b2a8545d0d4d98d994e0e2f23dff98fb15377e567faa0814791bda1275e3"}
Jul 07 16:25:33.685107 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:33.684713    2566 scope.go:117] "RemoveContainer" containerID="65c12247db59c5aee938b9789d4d297d13d380fa11448e60593099ff1e5e7680"
Jul 07 16:25:33.710137 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:33.710104    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-1cea82-openshift-default-6ff8898c79-7mshg"]
Jul 07 16:25:33.714155 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:33.714123    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-1cea82-openshift-default-6ff8898c79-7mshg"]
Jul 07 16:25:34.474656 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:34.474621    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f28c34bc-0591-4290-9d69-3f8a1cb95213" path="/var/lib/kubelet/pods/f28c34bc-0591-4290-9d69-3f8a1cb95213/volumes"
Jul 07 16:25:34.880222 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:34.880143    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-56b56968cc-gxml2_fde2c89a-f756-424d-9630-8993c5eee62c/authorino/0.log"
Jul 07 16:25:35.001943 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:35.001916    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-8dwtd_9cbfb56e-0a7a-4f1f-8eb8-df1615d4bc55/manager/0.log"
Jul 07 16:25:35.113326 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:35.113300    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-ng7j5_5320cfe0-479b-4ef9-bc2f-b3c368461783/manager/0.log"
Jul 07 16:25:35.340086 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:35.340054    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-8hf4r_d933ecaf-43d5-4993-9851-793a315835b2/registry-server/0.log"
Jul 07 16:25:35.458559 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:35.458527    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-n669q_38eba44e-f8d4-4e90-9173-ee3251128848/manager/0.log"
Jul 07 16:25:35.569943 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:35.569916    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-mk99s_c1c7a85b-891b-46ea-a556-8ab9e4f128f6/limitador/0.log"
Jul 07 16:25:35.686689 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:35.686607    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-lg2l4_080269f8-421c-4ca6-bc67-ad9b974c6ca2/manager/0.log"
Jul 07 16:25:36.384923 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:36.384853    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-59ffcfbcb9-n7k4p_18a0d440-92e1-41b8-91ab-b9bb82c6d229/kube-auth-proxy/0.log"
Jul 07 16:25:36.503000 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:36.502974    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-nxk9c_fc6066ec-a510-4f03-b98d-14c596d5b6ba/istio-proxy/1.log"
Jul 07 16:25:37.791591 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:37.791563    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-pc5ps_ccee4b1b-9e55-44b7-8374-5374d67a0fa3/main/0.log"
Jul 07 16:25:42.235520 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:42.235479    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-s9dw8/must-gather-rzpnf"]
Jul 07 16:25:42.236051 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:42.236026    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f28c34bc-0591-4290-9d69-3f8a1cb95213" containerName="istio-proxy"
Jul 07 16:25:42.236133 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:42.236056    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f28c34bc-0591-4290-9d69-3f8a1cb95213" containerName="istio-proxy"
Jul 07 16:25:42.236133 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:42.236075    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a61b604-4f78-4d6e-bf5c-11325716c60c" containerName="main"
Jul 07 16:25:42.236133 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:42.236086    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a61b604-4f78-4d6e-bf5c-11325716c60c" containerName="main"
Jul 07 16:25:42.236133 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:42.236108    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72882acd-bf9d-437a-8d6d-178506f8f24a" containerName="main"
Jul 07 16:25:42.236133 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:42.236116    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="72882acd-bf9d-437a-8d6d-178506f8f24a" containerName="main"
Jul 07 16:25:42.236133 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:42.236133    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f28c34bc-0591-4290-9d69-3f8a1cb95213" containerName="istio-proxy"
Jul 07 16:25:42.236434 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:42.236141    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f28c34bc-0591-4290-9d69-3f8a1cb95213" containerName="istio-proxy"
Jul 07 16:25:42.236434 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:42.236233    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="f28c34bc-0591-4290-9d69-3f8a1cb95213" containerName="istio-proxy"
Jul 07 16:25:42.236434 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:42.236249    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a61b604-4f78-4d6e-bf5c-11325716c60c" containerName="main"
Jul 07 16:25:42.236434 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:42.236259    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="72882acd-bf9d-437a-8d6d-178506f8f24a" containerName="main"
Jul 07 16:25:42.236619 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:42.236438    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="f28c34bc-0591-4290-9d69-3f8a1cb95213" containerName="istio-proxy"
Jul 07 16:25:42.241254 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:42.241229    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-s9dw8/must-gather-rzpnf"
Jul 07 16:25:42.245155 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:42.245134    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-s9dw8\"/\"kube-root-ca.crt\""
Jul 07 16:25:42.245749 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:42.245728    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-s9dw8\"/\"openshift-service-ca.crt\""
Jul 07 16:25:42.245875 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:42.245735    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-s9dw8\"/\"default-dockercfg-9fs8h\""
Jul 07 16:25:42.257289 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:42.257262    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-s9dw8/must-gather-rzpnf"]
Jul 07 16:25:42.340935 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:42.340870    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/42a3dbd1-89d2-402d-b50e-b95dc9dd27be-must-gather-output\") pod \"must-gather-rzpnf\" (UID: \"42a3dbd1-89d2-402d-b50e-b95dc9dd27be\") " pod="openshift-must-gather-s9dw8/must-gather-rzpnf"
Jul 07 16:25:42.341114 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:42.340995    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8k66\" (UniqueName: \"kubernetes.io/projected/42a3dbd1-89d2-402d-b50e-b95dc9dd27be-kube-api-access-n8k66\") pod \"must-gather-rzpnf\" (UID: \"42a3dbd1-89d2-402d-b50e-b95dc9dd27be\") " pod="openshift-must-gather-s9dw8/must-gather-rzpnf"
Jul 07 16:25:42.441374 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:42.441328    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8k66\" (UniqueName: \"kubernetes.io/projected/42a3dbd1-89d2-402d-b50e-b95dc9dd27be-kube-api-access-n8k66\") pod \"must-gather-rzpnf\" (UID: \"42a3dbd1-89d2-402d-b50e-b95dc9dd27be\") " pod="openshift-must-gather-s9dw8/must-gather-rzpnf"
Jul 07 16:25:42.441579 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:42.441433    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/42a3dbd1-89d2-402d-b50e-b95dc9dd27be-must-gather-output\") pod \"must-gather-rzpnf\" (UID: \"42a3dbd1-89d2-402d-b50e-b95dc9dd27be\") " pod="openshift-must-gather-s9dw8/must-gather-rzpnf"
Jul 07 16:25:42.441788 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:42.441765    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/42a3dbd1-89d2-402d-b50e-b95dc9dd27be-must-gather-output\") pod \"must-gather-rzpnf\" (UID: \"42a3dbd1-89d2-402d-b50e-b95dc9dd27be\") " pod="openshift-must-gather-s9dw8/must-gather-rzpnf"
Jul 07 16:25:42.451604 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:42.451579    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8k66\" (UniqueName: \"kubernetes.io/projected/42a3dbd1-89d2-402d-b50e-b95dc9dd27be-kube-api-access-n8k66\") pod \"must-gather-rzpnf\" (UID: \"42a3dbd1-89d2-402d-b50e-b95dc9dd27be\") " pod="openshift-must-gather-s9dw8/must-gather-rzpnf"
Jul 07 16:25:42.550857 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:42.550762    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-s9dw8/must-gather-rzpnf"
Jul 07 16:25:42.689191 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:42.689099    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-s9dw8/must-gather-rzpnf"]
Jul 07 16:25:42.692133 ip-10-0-138-101 kubenswrapper[2566]: W0707 16:25:42.692103    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod42a3dbd1_89d2_402d_b50e_b95dc9dd27be.slice/crio-604cd540222c8b7c5bea624781d6e4ecbe40e8b19444dad38e4c8c45355fd1d7 WatchSource:0}: Error finding container 604cd540222c8b7c5bea624781d6e4ecbe40e8b19444dad38e4c8c45355fd1d7: Status 404 returned error can't find the container with id 604cd540222c8b7c5bea624781d6e4ecbe40e8b19444dad38e4c8c45355fd1d7
Jul 07 16:25:42.694453 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:42.694438    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 16:25:42.732153 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:42.732121    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-s9dw8/must-gather-rzpnf" event={"ID":"42a3dbd1-89d2-402d-b50e-b95dc9dd27be","Type":"ContainerStarted","Data":"604cd540222c8b7c5bea624781d6e4ecbe40e8b19444dad38e4c8c45355fd1d7"}
Jul 07 16:25:43.743714 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:43.743660    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-s9dw8/must-gather-rzpnf" event={"ID":"42a3dbd1-89d2-402d-b50e-b95dc9dd27be","Type":"ContainerStarted","Data":"d43b597451d0a30464be5632072218150112acc757e71f32734f886b467a8660"}
Jul 07 16:25:44.753217 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:44.753162    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-s9dw8/must-gather-rzpnf" event={"ID":"42a3dbd1-89d2-402d-b50e-b95dc9dd27be","Type":"ContainerStarted","Data":"ce93a05964656d5268e3f03185d50ecdd9499b47224bc5bbacfaa6752b4b2be4"}
Jul 07 16:25:44.785768 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:44.785706    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-s9dw8/must-gather-rzpnf" podStartSLOduration=1.902911643 podStartE2EDuration="2.785682965s" totalImagesPullingTime="882.771322ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:25:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:25:44.784468451 +0000 UTC m=+2498.872894133" watchObservedRunningTime="2026-07-07 16:25:44.785682965 +0000 UTC m=+2498.874108638"
Jul 07 16:25:45.462701 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:45.462666    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-fvkrg_f0d07c9b-7fac-4665-b397-d996f6781c50/global-pull-secret-syncer/0.log"
Jul 07 16:25:45.563465 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:45.563432    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-r9nv8_4e92cac1-64ea-498d-b27c-30403f6e8b81/konnectivity-agent/0.log"
Jul 07 16:25:45.661777 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:45.661748    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-101.ec2.internal_9b6c8b6d42e5b52ab4cf9f706a9be0f7/haproxy/0.log"
Jul 07 16:25:51.238900 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:51.238841    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-56b56968cc-gxml2_fde2c89a-f756-424d-9630-8993c5eee62c/authorino/0.log"
Jul 07 16:25:51.278064 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:51.278027    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-8dwtd_9cbfb56e-0a7a-4f1f-8eb8-df1615d4bc55/manager/0.log"
Jul 07 16:25:51.322398 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:51.322366    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-ng7j5_5320cfe0-479b-4ef9-bc2f-b3c368461783/manager/0.log"
Jul 07 16:25:51.414189 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:51.414074    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-8hf4r_d933ecaf-43d5-4993-9851-793a315835b2/registry-server/0.log"
Jul 07 16:25:51.550747 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:51.550658    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-n669q_38eba44e-f8d4-4e90-9173-ee3251128848/manager/0.log"
Jul 07 16:25:51.576975 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:51.576930    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-mk99s_c1c7a85b-891b-46ea-a556-8ab9e4f128f6/limitador/0.log"
Jul 07 16:25:51.659907 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:51.659817    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-lg2l4_080269f8-421c-4ca6-bc67-ad9b974c6ca2/manager/0.log"
Jul 07 16:25:53.991356 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:53.991325    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-t4tv9_4c27c86b-6f34-4bb7-ae7b-1da713caf840/node-exporter/0.log"
Jul 07 16:25:54.018198 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:54.018152    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-t4tv9_4c27c86b-6f34-4bb7-ae7b-1da713caf840/kube-rbac-proxy/0.log"
Jul 07 16:25:54.046263 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:54.046224    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-t4tv9_4c27c86b-6f34-4bb7-ae7b-1da713caf840/init-textfile/0.log"
Jul 07 16:25:55.399488 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:55.399444    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-s9dw8/perf-node-gather-daemonset-npk8t"]
Jul 07 16:25:55.406508 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:55.406468    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-s9dw8/perf-node-gather-daemonset-npk8t"
Jul 07 16:25:55.423809 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:55.423777    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-s9dw8/perf-node-gather-daemonset-npk8t"]
Jul 07 16:25:55.473708 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:55.473673    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1ca46538-88d8-4ca0-91d8-0a886c7da4f0-lib-modules\") pod \"perf-node-gather-daemonset-npk8t\" (UID: \"1ca46538-88d8-4ca0-91d8-0a886c7da4f0\") " pod="openshift-must-gather-s9dw8/perf-node-gather-daemonset-npk8t"
Jul 07 16:25:55.473708 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:55.473723    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5v7vc\" (UniqueName: \"kubernetes.io/projected/1ca46538-88d8-4ca0-91d8-0a886c7da4f0-kube-api-access-5v7vc\") pod \"perf-node-gather-daemonset-npk8t\" (UID: \"1ca46538-88d8-4ca0-91d8-0a886c7da4f0\") " pod="openshift-must-gather-s9dw8/perf-node-gather-daemonset-npk8t"
Jul 07 16:25:55.474026 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:55.473759    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1ca46538-88d8-4ca0-91d8-0a886c7da4f0-sys\") pod \"perf-node-gather-daemonset-npk8t\" (UID: \"1ca46538-88d8-4ca0-91d8-0a886c7da4f0\") " pod="openshift-must-gather-s9dw8/perf-node-gather-daemonset-npk8t"
Jul 07 16:25:55.474026 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:55.473787    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1ca46538-88d8-4ca0-91d8-0a886c7da4f0-proc\") pod \"perf-node-gather-daemonset-npk8t\" (UID: \"1ca46538-88d8-4ca0-91d8-0a886c7da4f0\") " pod="openshift-must-gather-s9dw8/perf-node-gather-daemonset-npk8t"
Jul 07 16:25:55.474026 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:55.473830    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1ca46538-88d8-4ca0-91d8-0a886c7da4f0-podres\") pod \"perf-node-gather-daemonset-npk8t\" (UID: \"1ca46538-88d8-4ca0-91d8-0a886c7da4f0\") " pod="openshift-must-gather-s9dw8/perf-node-gather-daemonset-npk8t"
Jul 07 16:25:55.575198 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:55.575160    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1ca46538-88d8-4ca0-91d8-0a886c7da4f0-podres\") pod \"perf-node-gather-daemonset-npk8t\" (UID: \"1ca46538-88d8-4ca0-91d8-0a886c7da4f0\") " pod="openshift-must-gather-s9dw8/perf-node-gather-daemonset-npk8t"
Jul 07 16:25:55.575409 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:55.575282    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1ca46538-88d8-4ca0-91d8-0a886c7da4f0-lib-modules\") pod \"perf-node-gather-daemonset-npk8t\" (UID: \"1ca46538-88d8-4ca0-91d8-0a886c7da4f0\") " pod="openshift-must-gather-s9dw8/perf-node-gather-daemonset-npk8t"
Jul 07 16:25:55.575409 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:55.575323    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v7vc\" (UniqueName: \"kubernetes.io/projected/1ca46538-88d8-4ca0-91d8-0a886c7da4f0-kube-api-access-5v7vc\") pod \"perf-node-gather-daemonset-npk8t\" (UID: \"1ca46538-88d8-4ca0-91d8-0a886c7da4f0\") " pod="openshift-must-gather-s9dw8/perf-node-gather-daemonset-npk8t"
Jul 07 16:25:55.575409 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:55.575361    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1ca46538-88d8-4ca0-91d8-0a886c7da4f0-sys\") pod \"perf-node-gather-daemonset-npk8t\" (UID: \"1ca46538-88d8-4ca0-91d8-0a886c7da4f0\") " pod="openshift-must-gather-s9dw8/perf-node-gather-daemonset-npk8t"
Jul 07 16:25:55.575409 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:55.575393    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1ca46538-88d8-4ca0-91d8-0a886c7da4f0-proc\") pod \"perf-node-gather-daemonset-npk8t\" (UID: \"1ca46538-88d8-4ca0-91d8-0a886c7da4f0\") " pod="openshift-must-gather-s9dw8/perf-node-gather-daemonset-npk8t"
Jul 07 16:25:55.575635 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:55.575495    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1ca46538-88d8-4ca0-91d8-0a886c7da4f0-proc\") pod \"perf-node-gather-daemonset-npk8t\" (UID: \"1ca46538-88d8-4ca0-91d8-0a886c7da4f0\") " pod="openshift-must-gather-s9dw8/perf-node-gather-daemonset-npk8t"
Jul 07 16:25:55.575689 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:55.575635    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1ca46538-88d8-4ca0-91d8-0a886c7da4f0-lib-modules\") pod \"perf-node-gather-daemonset-npk8t\" (UID: \"1ca46538-88d8-4ca0-91d8-0a886c7da4f0\") " pod="openshift-must-gather-s9dw8/perf-node-gather-daemonset-npk8t"
Jul 07 16:25:55.575689 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:55.575635    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1ca46538-88d8-4ca0-91d8-0a886c7da4f0-sys\") pod \"perf-node-gather-daemonset-npk8t\" (UID: \"1ca46538-88d8-4ca0-91d8-0a886c7da4f0\") " pod="openshift-must-gather-s9dw8/perf-node-gather-daemonset-npk8t"
Jul 07 16:25:55.575782 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:55.575709    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1ca46538-88d8-4ca0-91d8-0a886c7da4f0-podres\") pod \"perf-node-gather-daemonset-npk8t\" (UID: \"1ca46538-88d8-4ca0-91d8-0a886c7da4f0\") " pod="openshift-must-gather-s9dw8/perf-node-gather-daemonset-npk8t"
Jul 07 16:25:55.586395 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:55.586358    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5v7vc\" (UniqueName: \"kubernetes.io/projected/1ca46538-88d8-4ca0-91d8-0a886c7da4f0-kube-api-access-5v7vc\") pod \"perf-node-gather-daemonset-npk8t\" (UID: \"1ca46538-88d8-4ca0-91d8-0a886c7da4f0\") " pod="openshift-must-gather-s9dw8/perf-node-gather-daemonset-npk8t"
Jul 07 16:25:55.718953 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:55.718907    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-s9dw8/perf-node-gather-daemonset-npk8t"
Jul 07 16:25:55.920385 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:55.920349    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-s9dw8/perf-node-gather-daemonset-npk8t"]
Jul 07 16:25:55.924661 ip-10-0-138-101 kubenswrapper[2566]: W0707 16:25:55.924620    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod1ca46538_88d8_4ca0_91d8_0a886c7da4f0.slice/crio-1c523d9b426eddc223ef54f2df83941fd16466e3d4961add0587a02c1b56351d WatchSource:0}: Error finding container 1c523d9b426eddc223ef54f2df83941fd16466e3d4961add0587a02c1b56351d: Status 404 returned error can't find the container with id 1c523d9b426eddc223ef54f2df83941fd16466e3d4961add0587a02c1b56351d
Jul 07 16:25:56.062317 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:56.062288    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6bffdf9778-6qpwk_d66b1fe3-70c4-422d-b909-9555280a9d1c/networking-console-plugin/0.log"
Jul 07 16:25:56.837644 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:56.837602    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-s9dw8/perf-node-gather-daemonset-npk8t" event={"ID":"1ca46538-88d8-4ca0-91d8-0a886c7da4f0","Type":"ContainerStarted","Data":"1c3662f083c3ac384de35dae152657a0055f7450a9e5922f3100ed4a86bf3576"}
Jul 07 16:25:56.838072 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:56.837675    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-s9dw8/perf-node-gather-daemonset-npk8t" event={"ID":"1ca46538-88d8-4ca0-91d8-0a886c7da4f0","Type":"ContainerStarted","Data":"1c523d9b426eddc223ef54f2df83941fd16466e3d4961add0587a02c1b56351d"}
Jul 07 16:25:56.838072 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:56.837750    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-s9dw8/perf-node-gather-daemonset-npk8t"
Jul 07 16:25:56.855782 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:56.855715    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-s9dw8/perf-node-gather-daemonset-npk8t" podStartSLOduration=1.855693244 podStartE2EDuration="1.855693244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 16:25:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:25:56.854955069 +0000 UTC m=+2510.943380741" watchObservedRunningTime="2026-07-07 16:25:56.855693244 +0000 UTC m=+2510.944118914"
Jul 07 16:25:59.181738 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:59.181625    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-z47rg_5fc809cc-0c34-4431-be3d-1d08204edde6/dns/0.log"
Jul 07 16:25:59.211773 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:59.211745    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-z47rg_5fc809cc-0c34-4431-be3d-1d08204edde6/kube-rbac-proxy/0.log"
Jul 07 16:25:59.250646 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:59.250616    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6cgnn_22f1cba0-dd6e-4f5a-8d37-d83d1f938ace/dns-node-resolver/0.log"
Jul 07 16:25:59.923017 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:59.922968    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-77d7b5bb7d-rcmxv_760703f9-9ff6-4867-bef4-85d0cb8c0687/registry/0.log"
Jul 07 16:25:59.980574 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:25:59.980550    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-shpjq_ff08ea94-65cb-4ca1-a272-3de7627d10b8/node-ca/0.log"
Jul 07 16:26:01.282061 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:26:01.282030    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-59ffcfbcb9-n7k4p_18a0d440-92e1-41b8-91ab-b9bb82c6d229/kube-auth-proxy/0.log"
Jul 07 16:26:01.320381 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:26:01.320335    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-nxk9c_fc6066ec-a510-4f03-b98d-14c596d5b6ba/istio-proxy/0.log"
Jul 07 16:26:01.355412 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:26:01.355381    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-nxk9c_fc6066ec-a510-4f03-b98d-14c596d5b6ba/istio-proxy/1.log"
Jul 07 16:26:02.312261 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:26:02.312229    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-t4pz9_f12ca464-a7b0-4009-ba1f-dde12c17d1c3/serve-healthcheck-canary/0.log"
Jul 07 16:26:02.856449 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:26:02.856421    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-s9dw8/perf-node-gather-daemonset-npk8t"
Jul 07 16:26:02.976686 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:26:02.976648    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bggdc_b7a06c58-5567-40aa-b24e-c5f71943f570/kube-rbac-proxy/0.log"
Jul 07 16:26:03.004640 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:26:03.004613    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bggdc_b7a06c58-5567-40aa-b24e-c5f71943f570/exporter/0.log"
Jul 07 16:26:03.035914 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:26:03.035870    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bggdc_b7a06c58-5567-40aa-b24e-c5f71943f570/extractor/0.log"
Jul 07 16:26:05.640634 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:26:05.640593    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-28lvw_cdc471c4-bece-4f05-9a9c-cf110a5eabb6/manager/0.log"
Jul 07 16:26:05.816179 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:26:05.816150    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-b69vl_73a96c42-cb3f-4acf-adf4-31c21f32186c/manager/0.log"
Jul 07 16:26:05.950145 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:26:05.949970    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-nkdqs_e5962e3e-69f6-42f4-b642-7bac15a37ede/server/0.log"
Jul 07 16:26:06.047596 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:26:06.047561    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-kt87x_d1ce20f8-4500-4af3-b2e9-d5c6933df650/manager/0.log"
Jul 07 16:26:06.273184 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:26:06.273096    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-56f57b78d4-tbrcq_7b3312b8-7ba2-4102-b36f-3b0f58178f4e/manager/0.log"
Jul 07 16:26:06.299522 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:26:06.299491    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-56f57b78d4-tbrcq_7b3312b8-7ba2-4102-b36f-3b0f58178f4e/copy-manifests/0.log"
Jul 07 16:26:06.611034 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:26:06.611003    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-59ffcfbcb9-n7k4p_18a0d440-92e1-41b8-91ab-b9bb82c6d229/kube-auth-proxy/0.log"
Jul 07 16:26:06.634870 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:26:06.634835    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-nxk9c_fc6066ec-a510-4f03-b98d-14c596d5b6ba/istio-proxy/0.log"
Jul 07 16:26:06.675082 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:26:06.675045    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-nxk9c_fc6066ec-a510-4f03-b98d-14c596d5b6ba/istio-proxy/1.log"
Jul 07 16:26:08.185400 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:26:08.185312    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-59ffcfbcb9-n7k4p_18a0d440-92e1-41b8-91ab-b9bb82c6d229/kube-auth-proxy/0.log"
Jul 07 16:26:08.215230 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:26:08.215188    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-nxk9c_fc6066ec-a510-4f03-b98d-14c596d5b6ba/istio-proxy/0.log"
Jul 07 16:26:08.249132 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:26:08.249093    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-nxk9c_fc6066ec-a510-4f03-b98d-14c596d5b6ba/istio-proxy/1.log"
Jul 07 16:26:13.089463 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:26:13.089435    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-58tdv_8a9671f3-44c2-473a-a8df-fd41bd2f4235/migrator/0.log"
Jul 07 16:26:13.118361 ip-10-0-138-101 kubenswrapper[2566]: I0707 16:26:13.118319    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-58tdv_8a9671f3-44c2-473a-a8df-fd41bd2f4235/graceful-termination/0.log"