Jul 14 11:35:29.865459 ip-10-0-136-173 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 14 11:35:29.865471 ip-10-0-136-173 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 14 11:35:29.865478 ip-10-0-136-173 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 14 11:35:29.865756 ip-10-0-136-173 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 14 11:35:39.892299 ip-10-0-136-173 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 14 11:35:39.892313 ip-10-0-136-173 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 95e06e2dc2af4871b43431af9ecb2a7b --
Jul 14 11:37:27.234739 ip-10-0-136-173 systemd[1]: Starting Kubernetes Kubelet...
Jul 14 11:37:27.526013 ip-10-0-136-173 kubenswrapper[2551]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 14 11:37:27.526013 ip-10-0-136-173 kubenswrapper[2551]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 14 11:37:27.526013 ip-10-0-136-173 kubenswrapper[2551]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 14 11:37:27.526013 ip-10-0-136-173 kubenswrapper[2551]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 14 11:37:27.526013 ip-10-0-136-173 kubenswrapper[2551]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 14 11:37:27.526776 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.526646    2551 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 14 11:37:27.530600 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.530580    2551 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 14 11:37:27.530600 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.530600    2551 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 14 11:37:27.530673 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.530604    2551 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 14 11:37:27.530673 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.530608    2551 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 14 11:37:27.530673 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.530612    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 14 11:37:27.530673 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.530615    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 14 11:37:27.530673 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.530617    2551 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 14 11:37:27.530673 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.530620    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 14 11:37:27.530673 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.530623    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 14 11:37:27.530673 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.530626    2551 feature_gate.go:328] unrecognized feature gate: Example2
Jul 14 11:37:27.530673 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.530629    2551 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 14 11:37:27.530673 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.530633    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 14 11:37:27.530673 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.530636    2551 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 14 11:37:27.530673 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.530639    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 14 11:37:27.530673 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.530641    2551 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 14 11:37:27.530673 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.530644    2551 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 14 11:37:27.530673 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.530654    2551 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 14 11:37:27.530673 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.530657    2551 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 14 11:37:27.530673 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.530661    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 14 11:37:27.530673 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.530664    2551 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 14 11:37:27.530673 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.530667    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 14 11:37:27.530673 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.530669    2551 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 14 11:37:27.531149 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.530672    2551 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 14 11:37:27.531149 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.530675    2551 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 14 11:37:27.531149 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.530678    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 14 11:37:27.531149 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.530681    2551 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 14 11:37:27.531149 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.530684    2551 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 14 11:37:27.531149 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.530687    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 14 11:37:27.531149 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.530689    2551 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 14 11:37:27.531149 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.530693    2551 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 14 11:37:27.531149 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.530697    2551 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 14 11:37:27.531149 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.530700    2551 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 14 11:37:27.531149 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.530704    2551 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 14 11:37:27.531149 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.530707    2551 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 14 11:37:27.531149 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.530710    2551 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 14 11:37:27.531149 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.530713    2551 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 14 11:37:27.531149 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.530715    2551 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 14 11:37:27.531149 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.530718    2551 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 14 11:37:27.531149 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.530721    2551 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 14 11:37:27.531149 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.530723    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 14 11:37:27.531149 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.530726    2551 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 14 11:37:27.531623 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.530728    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 14 11:37:27.531623 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.530731    2551 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 14 11:37:27.531623 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.530733    2551 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 14 11:37:27.531623 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.530736    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 14 11:37:27.531623 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.530739    2551 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 14 11:37:27.531623 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.530742    2551 feature_gate.go:328] unrecognized feature gate: Example
Jul 14 11:37:27.531623 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.530744    2551 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 14 11:37:27.531623 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.530747    2551 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 14 11:37:27.531623 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.530750    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 14 11:37:27.531623 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.530753    2551 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 14 11:37:27.531623 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.530755    2551 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 14 11:37:27.531623 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.530757    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 14 11:37:27.531623 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.530761    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 14 11:37:27.531623 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.530764    2551 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 14 11:37:27.531623 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.530768    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 14 11:37:27.531623 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.530770    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 14 11:37:27.531623 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.530773    2551 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 14 11:37:27.531623 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.530776    2551 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 14 11:37:27.531623 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.530779    2551 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 14 11:37:27.531623 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.530782    2551 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 14 11:37:27.532116 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.530785    2551 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 14 11:37:27.532116 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.530787    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 14 11:37:27.532116 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.530790    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 14 11:37:27.532116 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.530802    2551 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 14 11:37:27.532116 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.530806    2551 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 14 11:37:27.532116 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.530809    2551 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 14 11:37:27.532116 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.530812    2551 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 14 11:37:27.532116 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.530815    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 14 11:37:27.532116 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.530818    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 14 11:37:27.532116 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.530821    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 14 11:37:27.532116 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.530824    2551 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 14 11:37:27.532116 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.530827    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 14 11:37:27.532116 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.530830    2551 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 14 11:37:27.532116 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.530833    2551 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 14 11:37:27.532116 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.530836    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 14 11:37:27.532116 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.530839    2551 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 14 11:37:27.532116 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.530842    2551 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 14 11:37:27.532116 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.530845    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 14 11:37:27.532116 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.530847    2551 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 14 11:37:27.532582 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.530850    2551 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 14 11:37:27.532582 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.530854    2551 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 14 11:37:27.532582 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.530858    2551 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 14 11:37:27.532582 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.530861    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 14 11:37:27.532582 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.530864    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 14 11:37:27.532582 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.530866    2551 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 14 11:37:27.532582 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.530869    2551 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 14 11:37:27.532582 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.530872    2551 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 14 11:37:27.532582 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.530875    2551 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 14 11:37:27.532582 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.530878    2551 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 14 11:37:27.532582 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.530880    2551 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 14 11:37:27.532582 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.530883    2551 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 14 11:37:27.532582 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.530885    2551 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 14 11:37:27.532582 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.530888    2551 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 14 11:37:27.532582 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.530890    2551 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 14 11:37:27.532582 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.530893    2551 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 14 11:37:27.532582 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.530896    2551 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 14 11:37:27.532582 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.530899    2551 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 14 11:37:27.532582 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.530901    2551 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 14 11:37:27.532582 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.531677    2551 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 14 11:37:27.533066 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.531686    2551 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 14 11:37:27.533066 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.531692    2551 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 14 11:37:27.533066 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.531696    2551 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 14 11:37:27.533066 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.531709    2551 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 14 11:37:27.533066 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.531714    2551 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 14 11:37:27.533066 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.531718    2551 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 14 11:37:27.533066 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.531725    2551 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 14 11:37:27.533066 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.531730    2551 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 14 11:37:27.533066 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.531734    2551 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 14 11:37:27.533066 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.531738    2551 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 14 11:37:27.533066 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.531742    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 14 11:37:27.533066 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.531746    2551 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 14 11:37:27.533066 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.531750    2551 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 14 11:37:27.533066 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.531753    2551 feature_gate.go:328] unrecognized feature gate: Example
Jul 14 11:37:27.533066 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.531757    2551 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 14 11:37:27.533066 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.531762    2551 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 14 11:37:27.533066 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.531766    2551 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 14 11:37:27.533066 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.531769    2551 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 14 11:37:27.533066 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.531775    2551 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 14 11:37:27.533545 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.531779    2551 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 14 11:37:27.533545 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.531784    2551 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 14 11:37:27.533545 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.531788    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 14 11:37:27.533545 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.531792    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 14 11:37:27.533545 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.531796    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 14 11:37:27.533545 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.531800    2551 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 14 11:37:27.533545 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.531805    2551 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 14 11:37:27.533545 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.531808    2551 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 14 11:37:27.533545 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.531812    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 14 11:37:27.533545 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.531816    2551 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 14 11:37:27.533545 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.531819    2551 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 14 11:37:27.533545 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.531823    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 14 11:37:27.533545 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.531826    2551 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 14 11:37:27.533545 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.531829    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 14 11:37:27.533545 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.531834    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 14 11:37:27.533545 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.531838    2551 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 14 11:37:27.533545 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.531842    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 14 11:37:27.533545 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.531846    2551 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 14 11:37:27.533545 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.531849    2551 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 14 11:37:27.534009 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.531855    2551 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 14 11:37:27.534009 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.531858    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 14 11:37:27.534009 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.531862    2551 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 14 11:37:27.534009 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.531866    2551 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 14 11:37:27.534009 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.531869    2551 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 14 11:37:27.534009 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.531873    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 14 11:37:27.534009 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.531876    2551 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 14 11:37:27.534009 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.531880    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 14 11:37:27.534009 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.531883    2551 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 14 11:37:27.534009 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.531887    2551 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 14 11:37:27.534009 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.531891    2551 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 14 11:37:27.534009 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.531895    2551 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 14 11:37:27.534009 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.531899    2551 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 14 11:37:27.534009 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.531903    2551 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 14 11:37:27.534009 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.531906    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 14 11:37:27.534009 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.531909    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 14 11:37:27.534009 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.531913    2551 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 14 11:37:27.534009 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.531917    2551 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 14 11:37:27.534009 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.531920    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 14 11:37:27.534009 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.531924    2551 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 14 11:37:27.534517 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.531927    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 14 11:37:27.534517 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.531931    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 14 11:37:27.534517 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.531934    2551 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 14 11:37:27.534517 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.531937    2551 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 14 11:37:27.534517 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.531941    2551 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 14 11:37:27.534517 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.531944    2551 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 14 11:37:27.534517 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.531948    2551 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 14 11:37:27.534517 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.531952    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 14 11:37:27.534517 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.531960    2551 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 14 11:37:27.534517 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.531964    2551 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 14 11:37:27.534517 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.531968    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 14 11:37:27.534517 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.531972    2551 feature_gate.go:328] unrecognized feature gate: Example2
Jul 14 11:37:27.534517 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.531975    2551 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 14 11:37:27.534517 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.531979    2551 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 14 11:37:27.534517 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.531982    2551 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 14 11:37:27.534517 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.531986    2551 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 14 11:37:27.534517 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.531989    2551 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 14 11:37:27.534517 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.531995    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 14 11:37:27.534517 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.531998    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 14 11:37:27.534992 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532014    2551 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 14 11:37:27.534992 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532018    2551 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 14 11:37:27.534992 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532021    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 14 11:37:27.534992 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532023    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 14 11:37:27.534992 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532027    2551 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 14 11:37:27.534992 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532030    2551 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 14 11:37:27.534992 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532032    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 14 11:37:27.534992 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532035    2551 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 14 11:37:27.534992 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532038    2551 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 14 11:37:27.534992 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532041    2551 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 14 11:37:27.534992 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532043    2551 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 14 11:37:27.534992 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532046    2551 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 14 11:37:27.534992 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532048    2551 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 14 11:37:27.534992 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532051    2551 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 14 11:37:27.534992 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532054    2551 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 14 11:37:27.534992 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532056    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 14 11:37:27.534992 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532059    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 14 11:37:27.534992 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532061    2551 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 14 11:37:27.534992 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532064    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 14 11:37:27.534992 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532066    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 14 11:37:27.535515 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532069    2551 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 14 11:37:27.535515 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532540    2551 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 14 11:37:27.535515 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532547    2551 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 14 11:37:27.535515 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532552    2551 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 14 11:37:27.535515 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532556    2551 feature_gate.go:328] unrecognized feature gate: Example2
Jul 14 11:37:27.535515 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532560    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 14 11:37:27.535515 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532563    2551 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 14 11:37:27.535515 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532566    2551 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 14 11:37:27.535515 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532569    2551 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 14 11:37:27.535515 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532572    2551 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 14 11:37:27.535515 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532575    2551 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 14 11:37:27.535515 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532578    2551 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 14 11:37:27.535515 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532581    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 14 11:37:27.535515 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532584    2551 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 14 11:37:27.535515 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532587    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 14 11:37:27.535515 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532590    2551 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 14 11:37:27.535515 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532594    2551 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 14 11:37:27.535515 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532598    2551 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 14 11:37:27.535515 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532600    2551 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 14 11:37:27.535968 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532603    2551 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 14 11:37:27.535968 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532606    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 14 11:37:27.535968 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532609    2551 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 14 11:37:27.535968 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532612    2551 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 14 11:37:27.535968 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532614    2551 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 14 11:37:27.535968 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532617    2551 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 14 11:37:27.535968 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532620    2551 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 14 11:37:27.535968 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532623    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 14 11:37:27.535968 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532626    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 14 11:37:27.535968 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532628    2551 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 14 11:37:27.535968 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532631    2551 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 14 11:37:27.535968 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532634    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 14 11:37:27.535968 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532637    2551 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 14 11:37:27.535968 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532639    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 14 11:37:27.535968 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532644    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 14 11:37:27.535968 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532647    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 14 11:37:27.535968 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532649    2551 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 14 11:37:27.535968 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532652    2551 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 14 11:37:27.535968 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532655    2551 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 14 11:37:27.536431 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532659    2551 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 14 11:37:27.536431 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532662    2551 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 14 11:37:27.536431 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532665    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 14 11:37:27.536431 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532668    2551 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 14 11:37:27.536431 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532671    2551 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 14 11:37:27.536431 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532674    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 14 11:37:27.536431 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532677    2551 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 14 11:37:27.536431 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532680    2551 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 14 11:37:27.536431 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532682    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 14 11:37:27.536431 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532685    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 14 11:37:27.536431 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532687    2551 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 14 11:37:27.536431 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532690    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 14 11:37:27.536431 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532693    2551 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 14 11:37:27.536431 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532696    2551 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 14 11:37:27.536431 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532698    2551 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 14 11:37:27.536431 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532701    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 14 11:37:27.536431 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532703    2551 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 14 11:37:27.536431 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532706    2551 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 14 11:37:27.536431 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532708    2551 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 14 11:37:27.536916 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532711    2551 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 14 11:37:27.536916 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532714    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 14 11:37:27.536916 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532716    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 14 11:37:27.536916 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532719    2551 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 14 11:37:27.536916 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532721    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 14 11:37:27.536916 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532724    2551 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 14 11:37:27.536916 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532726    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 14 11:37:27.536916 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532729    2551 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 14 11:37:27.536916 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532733    2551 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 14 11:37:27.536916 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532736    2551 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 14 11:37:27.536916 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532739    2551 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 14 11:37:27.536916 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532741    2551 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 14 11:37:27.536916 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532744    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 14 11:37:27.536916 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532747    2551 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 14 11:37:27.536916 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532749    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 14 11:37:27.536916 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532752    2551 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 14 11:37:27.536916 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532755    2551 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 14 11:37:27.536916 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532758    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 14 11:37:27.536916 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532761    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 14 11:37:27.536916 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532763    2551 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 14 11:37:27.537402 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532766    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 14 11:37:27.537402 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532768    2551 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 14 11:37:27.537402 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532771    2551 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 14 11:37:27.537402 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532777    2551 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 14 11:37:27.537402 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532779    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 14 11:37:27.537402 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532783    2551 feature_gate.go:328] unrecognized feature gate: Example
Jul 14 11:37:27.537402 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532785    2551 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 14 11:37:27.537402 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532787    2551 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 14 11:37:27.537402 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532791    2551 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 14 11:37:27.537402 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532794    2551 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 14 11:37:27.537402 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532796    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 14 11:37:27.537402 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532799    2551 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 14 11:37:27.537402 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532802    2551 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 14 11:37:27.537402 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532804    2551 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 14 11:37:27.537402 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532806    2551 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 14 11:37:27.537402 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532809    2551 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 14 11:37:27.537402 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532811    2551 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 14 11:37:27.537402 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532814    2551 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 14 11:37:27.537402 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532816    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 14 11:37:27.537402 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532819    2551 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 14 11:37:27.537878 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532823    2551 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 14 11:37:27.537878 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532826    2551 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 14 11:37:27.537878 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.532828    2551 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 14 11:37:27.537878 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.532902    2551 flags.go:64] FLAG: --address="0.0.0.0"
Jul 14 11:37:27.537878 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.532909    2551 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 14 11:37:27.537878 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.532916    2551 flags.go:64] FLAG: --anonymous-auth="true"
Jul 14 11:37:27.537878 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.532921    2551 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 14 11:37:27.537878 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.532926    2551 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 14 11:37:27.537878 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.532929    2551 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 14 11:37:27.537878 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.532934    2551 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 14 11:37:27.537878 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.532938    2551 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 14 11:37:27.537878 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.532941    2551 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 14 11:37:27.537878 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.532944    2551 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 14 11:37:27.537878 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.532948    2551 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 14 11:37:27.537878 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.532951    2551 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 14 11:37:27.537878 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.532954    2551 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 14 11:37:27.537878 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.532962    2551 flags.go:64] FLAG: --cgroup-root=""
Jul 14 11:37:27.537878 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.532966    2551 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 14 11:37:27.537878 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.532970    2551 flags.go:64] FLAG: --client-ca-file=""
Jul 14 11:37:27.537878 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.532973    2551 flags.go:64] FLAG: --cloud-provider="external"
Jul 14 11:37:27.537878 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.532976    2551 flags.go:64] FLAG: --cluster-dns="[]"
Jul 14 11:37:27.537878 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.532980    2551 flags.go:64] FLAG: --cluster-domain=""
Jul 14 11:37:27.537878 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.532983    2551 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 14 11:37:27.537878 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.532986    2551 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 14 11:37:27.538490 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.532989    2551 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 14 11:37:27.538490 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.532993    2551 flags.go:64] FLAG: --container-log-max-files="5"
Jul 14 11:37:27.538490 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.532996    2551 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 14 11:37:27.538490 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533000    2551 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 14 11:37:27.538490 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533003    2551 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 14 11:37:27.538490 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533006    2551 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 14 11:37:27.538490 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533010    2551 flags.go:64] FLAG: --contention-profiling="false"
Jul 14 11:37:27.538490 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533012    2551 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 14 11:37:27.538490 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533018    2551 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 14 11:37:27.538490 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533023    2551 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 14 11:37:27.538490 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533027    2551 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 14 11:37:27.538490 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533033    2551 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 14 11:37:27.538490 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533036    2551 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 14 11:37:27.538490 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533039    2551 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 14 11:37:27.538490 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533042    2551 flags.go:64] FLAG: --enable-load-reader="false"
Jul 14 11:37:27.538490 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533045    2551 flags.go:64] FLAG: --enable-server="true"
Jul 14 11:37:27.538490 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533048    2551 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 14 11:37:27.538490 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533053    2551 flags.go:64] FLAG: --event-burst="100"
Jul 14 11:37:27.538490 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533056    2551 flags.go:64] FLAG: --event-qps="50"
Jul 14 11:37:27.538490 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533059    2551 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 14 11:37:27.538490 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533063    2551 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 14 11:37:27.538490 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533066    2551 flags.go:64] FLAG: --eviction-hard=""
Jul 14 11:37:27.538490 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533070    2551 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 14 11:37:27.538490 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533073    2551 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 14 11:37:27.539063 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533078    2551 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 14 11:37:27.539063 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533081    2551 flags.go:64] FLAG: --eviction-soft=""
Jul 14 11:37:27.539063 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533084    2551 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 14 11:37:27.539063 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533087    2551 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 14 11:37:27.539063 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533090    2551 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 14 11:37:27.539063 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533093    2551 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 14 11:37:27.539063 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533096    2551 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 14 11:37:27.539063 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533099    2551 flags.go:64] FLAG: --fail-swap-on="true"
Jul 14 11:37:27.539063 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533102    2551 flags.go:64] FLAG: --feature-gates=""
Jul 14 11:37:27.539063 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533107    2551 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 14 11:37:27.539063 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533110    2551 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 14 11:37:27.539063 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533113    2551 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 14 11:37:27.539063 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533116    2551 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 14 11:37:27.539063 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533119    2551 flags.go:64] FLAG: --healthz-port="10248"
Jul 14 11:37:27.539063 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533122    2551 flags.go:64] FLAG: --help="false"
Jul 14 11:37:27.539063 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533125    2551 flags.go:64] FLAG: --hostname-override="ip-10-0-136-173.ec2.internal"
Jul 14 11:37:27.539063 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533129    2551 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 14 11:37:27.539063 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533132    2551 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 14 11:37:27.539063 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533135    2551 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 14 11:37:27.539063 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533139    2551 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 14 11:37:27.539063 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533142    2551 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 14 11:37:27.539063 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533145    2551 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 14 11:37:27.539063 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533148    2551 flags.go:64] FLAG: --image-service-endpoint=""
Jul 14 11:37:27.539860 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533151    2551 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 14 11:37:27.539860 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533154    2551 flags.go:64] FLAG: --kube-api-burst="100"
Jul 14 11:37:27.539860 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533157    2551 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 14 11:37:27.539860 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533160    2551 flags.go:64] FLAG: --kube-api-qps="50"
Jul 14 11:37:27.539860 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533163    2551 flags.go:64] FLAG: --kube-reserved=""
Jul 14 11:37:27.539860 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533166    2551 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 14 11:37:27.539860 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533169    2551 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 14 11:37:27.539860 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533172    2551 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 14 11:37:27.539860 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533175    2551 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 14 11:37:27.539860 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533179    2551 flags.go:64] FLAG: --lock-file=""
Jul 14 11:37:27.539860 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533182    2551 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 14 11:37:27.539860 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533185    2551 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 14 11:37:27.539860 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533188    2551 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 14 11:37:27.539860 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533194    2551 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 14 11:37:27.539860 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533197    2551 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 14 11:37:27.539860 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533200    2551 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 14 11:37:27.539860 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533203    2551 flags.go:64] FLAG: --logging-format="text"
Jul 14 11:37:27.539860 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533206    2551 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 14 11:37:27.539860 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533209    2551 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 14 11:37:27.539860 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533212    2551 flags.go:64] FLAG: --manifest-url=""
Jul 14 11:37:27.539860 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533215    2551 flags.go:64] FLAG: --manifest-url-header=""
Jul 14 11:37:27.539860 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533219    2551 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 14 11:37:27.539860 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533222    2551 flags.go:64] FLAG: --max-open-files="1000000"
Jul 14 11:37:27.539860 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533226    2551 flags.go:64] FLAG: --max-pods="110"
Jul 14 11:37:27.539860 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533229    2551 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 14 11:37:27.541012 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533233    2551 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 14 11:37:27.541012 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533236    2551 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 14 11:37:27.541012 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533239    2551 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 14 11:37:27.541012 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533242    2551 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 14 11:37:27.541012 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533245    2551 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 14 11:37:27.541012 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533248    2551 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 14 11:37:27.541012 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533256    2551 flags.go:64] FLAG: --node-status-max-images="50"
Jul 14 11:37:27.541012 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533259    2551 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 14 11:37:27.541012 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533262    2551 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 14 11:37:27.541012 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533265    2551 flags.go:64] FLAG: --pod-cidr=""
Jul 14 11:37:27.541012 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533268    2551 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926"
Jul 14 11:37:27.541012 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533275    2551 flags.go:64] FLAG: --pod-manifest-path=""
Jul 14 11:37:27.541012 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533278    2551 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 14 11:37:27.541012 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533281    2551 flags.go:64] FLAG: --pods-per-core="0"
Jul 14 11:37:27.541012 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533284    2551 flags.go:64] FLAG: --port="10250"
Jul 14 11:37:27.541012 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533287    2551 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 14 11:37:27.541012 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533291    2551 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0da5025ee40b813c3"
Jul 14 11:37:27.541012 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533295    2551 flags.go:64] FLAG: --qos-reserved=""
Jul 14 11:37:27.541012 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533297    2551 flags.go:64] FLAG: --read-only-port="10255"
Jul 14 11:37:27.541012 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533300    2551 flags.go:64] FLAG: --register-node="true"
Jul 14 11:37:27.541012 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533303    2551 flags.go:64] FLAG: --register-with-taints=""
Jul 14 11:37:27.541012 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533307    2551 flags.go:64] FLAG: --registry-burst="10"
Jul 14 11:37:27.541012 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533310    2551 flags.go:64] FLAG: --registry-qps="5"
Jul 14 11:37:27.541012 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533313    2551 flags.go:64] FLAG: --reserved-cpus=""
Jul 14 11:37:27.541941 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533317    2551 flags.go:64] FLAG: --reserved-memory=""
Jul 14 11:37:27.541941 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533320    2551 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 14 11:37:27.541941 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533324    2551 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 14 11:37:27.541941 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533327    2551 flags.go:64] FLAG: --rotate-certificates="false"
Jul 14 11:37:27.541941 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533330    2551 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 14 11:37:27.541941 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533333    2551 flags.go:64] FLAG: --runonce="false"
Jul 14 11:37:27.541941 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533335    2551 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 14 11:37:27.541941 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533339    2551 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 14 11:37:27.541941 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533341    2551 flags.go:64] FLAG: --seccomp-default="false"
Jul 14 11:37:27.541941 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533346    2551 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 14 11:37:27.541941 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533349    2551 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 14 11:37:27.541941 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533352    2551 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 14 11:37:27.541941 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533355    2551 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 14 11:37:27.541941 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533358    2551 flags.go:64] FLAG: --storage-driver-password="root"
Jul 14 11:37:27.541941 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533361    2551 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 14 11:37:27.541941 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533364    2551 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 14 11:37:27.541941 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533367    2551 flags.go:64] FLAG: --storage-driver-user="root"
Jul 14 11:37:27.541941 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533370    2551 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 14 11:37:27.541941 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533373    2551 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 14 11:37:27.541941 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533376    2551 flags.go:64] FLAG: --system-cgroups=""
Jul 14 11:37:27.541941 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533379    2551 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 14 11:37:27.541941 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533385    2551 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 14 11:37:27.541941 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533400    2551 flags.go:64] FLAG: --tls-cert-file=""
Jul 14 11:37:27.541941 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533403    2551 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 14 11:37:27.541941 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533409    2551 flags.go:64] FLAG: --tls-min-version=""
Jul 14 11:37:27.542571 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533412    2551 flags.go:64] FLAG: --tls-private-key-file=""
Jul 14 11:37:27.542571 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533415    2551 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 14 11:37:27.542571 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533418    2551 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 14 11:37:27.542571 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533421    2551 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 14 11:37:27.542571 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533425    2551 flags.go:64] FLAG: --v="2"
Jul 14 11:37:27.542571 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533429    2551 flags.go:64] FLAG: --version="false"
Jul 14 11:37:27.542571 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533433    2551 flags.go:64] FLAG: --vmodule=""
Jul 14 11:37:27.542571 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533437    2551 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 14 11:37:27.542571 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.533441    2551 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 14 11:37:27.542571 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.533561    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 14 11:37:27.542571 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.533565    2551 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 14 11:37:27.542571 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.533568    2551 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 14 11:37:27.542571 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.533571    2551 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 14 11:37:27.542571 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.533573    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 14 11:37:27.542571 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.533576    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 14 11:37:27.542571 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.533579    2551 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 14 11:37:27.542571 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.533582    2551 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 14 11:37:27.542571 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.533585    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 14 11:37:27.542571 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.533587    2551 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 14 11:37:27.542571 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.533590    2551 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 14 11:37:27.542571 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.533592    2551 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 14 11:37:27.542571 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.533595    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 14 11:37:27.543154 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.533597    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 14 11:37:27.543154 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.533600    2551 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 14 11:37:27.543154 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.533602    2551 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 14 11:37:27.543154 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.533605    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 14 11:37:27.543154 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.533607    2551 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 14 11:37:27.543154 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.533610    2551 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 14 11:37:27.543154 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.533612    2551 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 14 11:37:27.543154 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.533615    2551 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 14 11:37:27.543154 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.533617    2551 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 14 11:37:27.543154 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.533621    2551 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 14 11:37:27.543154 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.533624    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 14 11:37:27.543154 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.533626    2551 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 14 11:37:27.543154 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.533629    2551 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 14 11:37:27.543154 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.533632    2551 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 14 11:37:27.543154 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.533634    2551 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 14 11:37:27.543154 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.533637    2551 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 14 11:37:27.543154 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.533640    2551 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 14 11:37:27.543154 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.533643    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 14 11:37:27.543154 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.533645    2551 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 14 11:37:27.543154 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.533647    2551 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 14 11:37:27.543664 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.533650    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 14 11:37:27.543664 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.533653    2551 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 14 11:37:27.543664 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.533655    2551 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 14 11:37:27.543664 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.533658    2551 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 14 11:37:27.543664 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.533660    2551 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 14 11:37:27.543664 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.533663    2551 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 14 11:37:27.543664 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.533665    2551 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 14 11:37:27.543664 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.533668    2551 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 14 11:37:27.543664 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.533670    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 14 11:37:27.543664 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.533672    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 14 11:37:27.543664 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.533675    2551 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 14 11:37:27.543664 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.533677    2551 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 14 11:37:27.543664 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.533680    2551 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 14 11:37:27.543664 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.533682    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 14 11:37:27.543664 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.533684    2551 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 14 11:37:27.543664 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.533687    2551 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 14 11:37:27.543664 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.533689    2551 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 14 11:37:27.543664 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.533692    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 14 11:37:27.543664 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.533694    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 14 11:37:27.543664 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.533697    2551 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 14 11:37:27.544148 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.533699    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 14 11:37:27.544148 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.533703    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 14 11:37:27.544148 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.533705    2551 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 14 11:37:27.544148 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.533708    2551 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 14 11:37:27.544148 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.533710    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 14 11:37:27.544148 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.533713    2551 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 14 11:37:27.544148 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.533715    2551 feature_gate.go:328] unrecognized feature gate: Example2
Jul 14 11:37:27.544148 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.533718    2551 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 14 11:37:27.544148 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.533720    2551 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 14 11:37:27.544148 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.533723    2551 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 14 11:37:27.544148 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.533727    2551 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 14 11:37:27.544148 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.533730    2551 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 14 11:37:27.544148 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.533733    2551 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 14 11:37:27.544148 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.533736    2551 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 14 11:37:27.544148 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.533739    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 14 11:37:27.544148 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.533742    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 14 11:37:27.544148 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.533745    2551 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 14 11:37:27.544148 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.533747    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 14 11:37:27.544148 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.533750    2551 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 14 11:37:27.544148 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.533754    2551 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 14 11:37:27.544653 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.533756    2551 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 14 11:37:27.544653 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.533759    2551 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 14 11:37:27.544653 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.533762    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 14 11:37:27.544653 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.533764    2551 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 14 11:37:27.544653 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.533767    2551 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 14 11:37:27.544653 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.533771    2551 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 14 11:37:27.544653 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.533774    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 14 11:37:27.544653 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.533777    2551 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 14 11:37:27.544653 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.533779    2551 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 14 11:37:27.544653 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.533782    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 14 11:37:27.544653 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.533784    2551 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 14 11:37:27.544653 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.533787    2551 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 14 11:37:27.544653 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.533789    2551 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 14 11:37:27.544653 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.533793    2551 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 14 11:37:27.544653 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.533795    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 14 11:37:27.544653 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.533799    2551 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 14 11:37:27.544653 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.533802    2551 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 14 11:37:27.544653 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.533805    2551 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 14 11:37:27.544653 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.533808    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 14 11:37:27.545110 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.533811    2551 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 14 11:37:27.545110 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.533814    2551 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 14 11:37:27.545110 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.533816    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 14 11:37:27.545110 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.533819    2551 feature_gate.go:328] unrecognized feature gate: Example
Jul 14 11:37:27.545110 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.533822    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 14 11:37:27.545110 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.533824    2551 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 14 11:37:27.545110 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.533827    2551 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 14 11:37:27.545110 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.534325    2551 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 14 11:37:27.545110 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.542994    2551 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 14 11:37:27.545110 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.543012    2551 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 14 11:37:27.545110 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.543033    2551 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 14 11:37:27.545110 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.543040    2551 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 14 11:37:27.545110 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543102    2551 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 14 11:37:27.545110 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543108    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 14 11:37:27.545110 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543111    2551 feature_gate.go:328] unrecognized feature gate: Example
Jul 14 11:37:27.545500 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543115    2551 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 14 11:37:27.545500 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543118    2551 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 14 11:37:27.545500 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543121    2551 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 14 11:37:27.545500 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543124    2551 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 14 11:37:27.545500 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543127    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 14 11:37:27.545500 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543130    2551 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 14 11:37:27.545500 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543133    2551 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 14 11:37:27.545500 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543136    2551 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 14 11:37:27.545500 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543138    2551 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 14 11:37:27.545500 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543141    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 14 11:37:27.545500 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543144    2551 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 14 11:37:27.545500 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543146    2551 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 14 11:37:27.545500 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543149    2551 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 14 11:37:27.545500 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543151    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 14 11:37:27.545500 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543154    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 14 11:37:27.545500 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543157    2551 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 14 11:37:27.545500 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543160    2551 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 14 11:37:27.545500 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543162    2551 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 14 11:37:27.545500 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543165    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 14 11:37:27.545500 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543168    2551 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 14 11:37:27.545974 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543171    2551 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 14 11:37:27.545974 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543173    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 14 11:37:27.545974 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543176    2551 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 14 11:37:27.545974 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543179    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 14 11:37:27.545974 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543182    2551 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 14 11:37:27.545974 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543185    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 14 11:37:27.545974 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543188    2551 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 14 11:37:27.545974 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543191    2551 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 14 11:37:27.545974 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543194    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 14 11:37:27.545974 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543196    2551 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 14 11:37:27.545974 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543200    2551 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 14 11:37:27.545974 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543205    2551 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 14 11:37:27.545974 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543208    2551 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 14 11:37:27.545974 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543212    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 14 11:37:27.545974 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543215    2551 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 14 11:37:27.545974 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543217    2551 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 14 11:37:27.545974 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543220    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 14 11:37:27.545974 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543222    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 14 11:37:27.545974 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543225    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 14 11:37:27.546436 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543228    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 14 11:37:27.546436 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543230    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 14 11:37:27.546436 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543233    2551 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 14 11:37:27.546436 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543235    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 14 11:37:27.546436 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543238    2551 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 14 11:37:27.546436 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543240    2551 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 14 11:37:27.546436 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543243    2551 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 14 11:37:27.546436 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543246    2551 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 14 11:37:27.546436 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543249    2551 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 14 11:37:27.546436 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543252    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 14 11:37:27.546436 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543255    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 14 11:37:27.546436 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543257    2551 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 14 11:37:27.546436 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543260    2551 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 14 11:37:27.546436 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543262    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 14 11:37:27.546436 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543265    2551 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 14 11:37:27.546436 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543267    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 14 11:37:27.546436 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543270    2551 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 14 11:37:27.546436 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543272    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 14 11:37:27.546436 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543275    2551 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 14 11:37:27.546436 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543278    2551 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 14 11:37:27.546924 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543281    2551 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 14 11:37:27.546924 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543284    2551 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 14 11:37:27.546924 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543287    2551 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 14 11:37:27.546924 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543291    2551 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 14 11:37:27.546924 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543295    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 14 11:37:27.546924 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543298    2551 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 14 11:37:27.546924 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543300    2551 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 14 11:37:27.546924 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543303    2551 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 14 11:37:27.546924 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543306    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 14 11:37:27.546924 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543309    2551 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 14 11:37:27.546924 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543311    2551 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 14 11:37:27.546924 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543313    2551 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 14 11:37:27.546924 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543316    2551 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 14 11:37:27.546924 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543319    2551 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 14 11:37:27.546924 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543322    2551 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 14 11:37:27.546924 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543324    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 14 11:37:27.546924 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543327    2551 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 14 11:37:27.546924 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543335    2551 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 14 11:37:27.546924 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543338    2551 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 14 11:37:27.547377 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543341    2551 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 14 11:37:27.547377 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543343    2551 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 14 11:37:27.547377 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543346    2551 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 14 11:37:27.547377 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543348    2551 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 14 11:37:27.547377 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543351    2551 feature_gate.go:328] unrecognized feature gate: Example2
Jul 14 11:37:27.547377 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543353    2551 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 14 11:37:27.547377 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543356    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 14 11:37:27.547377 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543359    2551 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 14 11:37:27.547377 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543361    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 14 11:37:27.547377 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543363    2551 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 14 11:37:27.547377 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543366    2551 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 14 11:37:27.547377 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543369    2551 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 14 11:37:27.547377 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543372    2551 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 14 11:37:27.547377 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543374    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 14 11:37:27.547377 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543377    2551 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 14 11:37:27.547377 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543380    2551 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 14 11:37:27.547377 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543383    2551 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 14 11:37:27.547377 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543387    2551 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 14 11:37:27.547842 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.543409    2551 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 14 11:37:27.547842 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543537    2551 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 14 11:37:27.547842 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543542    2551 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 14 11:37:27.547842 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543546    2551 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 14 11:37:27.547842 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543548    2551 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 14 11:37:27.547842 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543551    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 14 11:37:27.547842 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543554    2551 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 14 11:37:27.547842 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543557    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 14 11:37:27.547842 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543560    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 14 11:37:27.547842 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543562    2551 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 14 11:37:27.547842 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543565    2551 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 14 11:37:27.547842 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543567    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 14 11:37:27.547842 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543570    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 14 11:37:27.547842 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543574    2551 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 14 11:37:27.547842 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543576    2551 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 14 11:37:27.548212 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543579    2551 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 14 11:37:27.548212 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543582    2551 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 14 11:37:27.548212 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543584    2551 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 14 11:37:27.548212 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543587    2551 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 14 11:37:27.548212 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543589    2551 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 14 11:37:27.548212 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543592    2551 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 14 11:37:27.548212 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543594    2551 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 14 11:37:27.548212 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543597    2551 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 14 11:37:27.548212 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543600    2551 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 14 11:37:27.548212 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543602    2551 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 14 11:37:27.548212 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543605    2551 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 14 11:37:27.548212 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543608    2551 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 14 11:37:27.548212 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543610    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 14 11:37:27.548212 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543614    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 14 11:37:27.548212 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543617    2551 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 14 11:37:27.548212 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543619    2551 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 14 11:37:27.548212 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543622    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 14 11:37:27.548212 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543625    2551 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 14 11:37:27.548212 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543627    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 14 11:37:27.548212 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543630    2551 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 14 11:37:27.548705 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543632    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 14 11:37:27.548705 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543635    2551 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 14 11:37:27.548705 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543637    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 14 11:37:27.548705 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543640    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 14 11:37:27.548705 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543642    2551 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 14 11:37:27.548705 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543645    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 14 11:37:27.548705 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543647    2551 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 14 11:37:27.548705 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543650    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 14 11:37:27.548705 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543654    2551 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 14 11:37:27.548705 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543657    2551 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 14 11:37:27.548705 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543660    2551 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 14 11:37:27.548705 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543664    2551 feature_gate.go:328] unrecognized feature gate: Example
Jul 14 11:37:27.548705 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543667    2551 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 14 11:37:27.548705 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543670    2551 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 14 11:37:27.548705 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543673    2551 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 14 11:37:27.548705 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543676    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 14 11:37:27.548705 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543680    2551 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 14 11:37:27.548705 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543683    2551 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 14 11:37:27.548705 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543686    2551 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 14 11:37:27.549192 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543689    2551 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 14 11:37:27.549192 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543692    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 14 11:37:27.549192 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543694    2551 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 14 11:37:27.549192 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543697    2551 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 14 11:37:27.549192 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543700    2551 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 14 11:37:27.549192 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543703    2551 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 14 11:37:27.549192 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543706    2551 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 14 11:37:27.549192 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543708    2551 feature_gate.go:328] unrecognized feature gate: Example2
Jul 14 11:37:27.549192 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543711    2551 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 14 11:37:27.549192 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543713    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 14 11:37:27.549192 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543715    2551 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 14 11:37:27.549192 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543718    2551 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 14 11:37:27.549192 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543721    2551 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 14 11:37:27.549192 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543723    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 14 11:37:27.549192 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543726    2551 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 14 11:37:27.549192 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543728    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 14 11:37:27.549192 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543731    2551 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 14 11:37:27.549192 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543733    2551 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 14 11:37:27.549192 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543736    2551 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 14 11:37:27.549192 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543738    2551 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 14 11:37:27.549770 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543741    2551 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 14 11:37:27.549770 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543744    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 14 11:37:27.549770 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543746    2551 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 14 11:37:27.549770 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543749    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 14 11:37:27.549770 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543751    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 14 11:37:27.549770 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543754    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 14 11:37:27.549770 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543757    2551 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 14 11:37:27.549770 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543761    2551 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 14 11:37:27.549770 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543764    2551 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 14 11:37:27.549770 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543767    2551 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 14 11:37:27.549770 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543770    2551 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 14 11:37:27.549770 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543772    2551 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 14 11:37:27.549770 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543775    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 14 11:37:27.549770 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543778    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 14 11:37:27.549770 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543780    2551 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 14 11:37:27.549770 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543783    2551 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 14 11:37:27.549770 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543786    2551 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 14 11:37:27.549770 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543789    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 14 11:37:27.549770 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543792    2551 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 14 11:37:27.549770 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543794    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 14 11:37:27.550272 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543816    2551 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 14 11:37:27.550272 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543821    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 14 11:37:27.550272 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543824    2551 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 14 11:37:27.550272 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543827    2551 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 14 11:37:27.550272 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543830    2551 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 14 11:37:27.550272 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:27.543832    2551 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 14 11:37:27.550272 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.543837    2551 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 14 11:37:27.550272 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.543973    2551 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 14 11:37:27.550272 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.545981    2551 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 14 11:37:27.550272 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.547055    2551 server.go:1019] "Starting client certificate rotation"
Jul 14 11:37:27.550272 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.547161    2551 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 14 11:37:27.550272 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.547232    2551 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 14 11:37:27.550272 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.549066    2551 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 14 11:37:27.552917 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.552896    2551 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 14 11:37:27.559093 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.559075    2551 log.go:25] "Validated CRI v1 runtime API"
Jul 14 11:37:27.563376 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.563360    2551 log.go:25] "Validated CRI v1 image API"
Jul 14 11:37:27.564323 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.564303    2551 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 14 11:37:27.567974 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.567947    2551 fs.go:135] Filesystem UUIDs: map[0255b66d-7834-4389-9942-c9385b1ed489:/dev/nvme0n1p3 312cd045-1ec2-4c26-b6c9-6e412a510528:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Jul 14 11:37:27.568065 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.567977    2551 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 14 11:37:27.575586 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.575443    2551 manager.go:216] Machine: {Timestamp:2026-07-14 11:37:27.574297549 +0000 UTC m=+0.250294024 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099438 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2a8774e50bb390a7f13b12c6f310fe SystemUUID:ec2a8774-e50b-b390-a7f1-3b12c6f310fe BootID:95e06e2d-c2af-4871-b434-31af9ecb2a7b Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:02:e7:9d:e8:29 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:02:e7:9d:e8:29 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:a6:c2:a2:53:bf:97 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 14 11:37:27.575586 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.575560    2551 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 14 11:37:27.575781 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.575689    2551 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 14 11:37:27.577538 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.577518    2551 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 14 11:37:27.577618 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.577489    2551 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 14 11:37:27.577761 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.577599    2551 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-173.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 14 11:37:27.578606 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.578596    2551 topology_manager.go:138] "Creating topology manager with none policy"
Jul 14 11:37:27.578650 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.578609    2551 container_manager_linux.go:309] "Creating device plugin manager"
Jul 14 11:37:27.578650 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.578617    2551 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 14 11:37:27.579132 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.579122    2551 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 14 11:37:27.580247 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.580236    2551 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 14 11:37:27.583707 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.583694    2551 state_mem.go:36] "Initialized new in-memory state store"
Jul 14 11:37:27.583833 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.583823    2551 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 14 11:37:27.583864 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.583853    2551 kubelet.go:486] "Attempting to sync node with API server"
Jul 14 11:37:27.583902 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.583865    2551 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 14 11:37:27.583902 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.583890    2551 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 14 11:37:27.583902 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.583903    2551 kubelet.go:398] "Adding apiserver pod source"
Jul 14 11:37:27.584009 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.583913    2551 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 14 11:37:27.584308 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.584296    2551 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 14 11:37:27.584348 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.584314    2551 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 14 11:37:27.586373 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.586353    2551 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 14 11:37:27.588508 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.588490    2551 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 14 11:37:27.588585 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.588522    2551 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 14 11:37:27.588707 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.588696    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 14 11:37:27.588738 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.588718    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 14 11:37:27.588738 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.588728    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 14 11:37:27.588738 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.588734    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 14 11:37:27.588811 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.588744    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 14 11:37:27.588811 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.588750    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 14 11:37:27.589049 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.589041    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 14 11:37:27.589077 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.589050    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 14 11:37:27.589077 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.589059    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 14 11:37:27.589077 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.589065    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 14 11:37:27.589148 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.589079    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 14 11:37:27.589148 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.589088    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 14 11:37:27.590097 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.590083    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 14 11:37:27.590097 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.590094    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 14 11:37:27.593313 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.593299    2551 server.go:1268] "Started kubelet"
Jul 14 11:37:27.593424 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.593381    2551 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 14 11:37:27.593600 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.593534    2551 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 14 11:37:27.593686 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.593626    2551 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 14 11:37:27.594304 ip-10-0-136-173 systemd[1]: Started Kubernetes Kubelet.
Jul 14 11:37:27.595234 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.595207    2551 server.go:310] "Adding debug handlers to kubelet server"
Jul 14 11:37:27.596978 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.596936    2551 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-173.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 14 11:37:27.596978 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:27.596942    2551 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 14 11:37:27.597141 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:27.596991    2551 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-173.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 14 11:37:27.597635 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.597615    2551 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 14 11:37:27.598926 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:27.598907    2551 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 14 11:37:27.601930 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.601910    2551 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 14 11:37:27.601930 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.601909    2551 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 14 11:37:27.602089 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:27.602048    2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-173.ec2.internal\" not found"
Jul 14 11:37:27.602564 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.602536    2551 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 14 11:37:27.602564 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.602538    2551 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 14 11:37:27.602678 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.602571    2551 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 14 11:37:27.602731 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.602716    2551 reconstruct.go:97] "Volume reconstruction finished"
Jul 14 11:37:27.602731 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.602729    2551 reconciler.go:29] "Reconciler: start to sync state"
Jul 14 11:37:27.603300 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:27.602614    2551 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-173.ec2.internal.18c224f772c67d94  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-173.ec2.internal,UID:ip-10-0-136-173.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-173.ec2.internal,},FirstTimestamp:2026-07-14 11:37:27.593270676 +0000 UTC m=+0.269267152,LastTimestamp:2026-07-14 11:37:27.593270676 +0000 UTC m=+0.269267152,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-173.ec2.internal,}"
Jul 14 11:37:27.603584 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.603570    2551 factory.go:55] Registering systemd factory
Jul 14 11:37:27.603624 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.603598    2551 factory.go:223] Registration of the systemd container factory successfully
Jul 14 11:37:27.603826 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.603812    2551 factory.go:153] Registering CRI-O factory
Jul 14 11:37:27.603826 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.603829    2551 factory.go:223] Registration of the crio container factory successfully
Jul 14 11:37:27.603901 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.603883    2551 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 14 11:37:27.603940 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.603909    2551 factory.go:103] Registering Raw factory
Jul 14 11:37:27.603940 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.603921    2551 manager.go:1196] Started watching for new ooms in manager
Jul 14 11:37:27.604500 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.604485    2551 manager.go:349] Starting recovery of all containers
Jul 14 11:37:27.607332 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:27.607299    2551 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 14 11:37:27.607432 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:27.607369    2551 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-173.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 14 11:37:27.612925 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.612880    2551 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 14 11:37:27.618128 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.618111    2551 manager.go:354] Recovery completed
Jul 14 11:37:27.622559 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.622543    2551 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 14 11:37:27.624494 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.624475    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-173.ec2.internal" event="NodeHasSufficientMemory"
Jul 14 11:37:27.624569 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.624507    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-173.ec2.internal" event="NodeHasNoDiskPressure"
Jul 14 11:37:27.624569 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.624519    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-173.ec2.internal" event="NodeHasSufficientPID"
Jul 14 11:37:27.625063 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.625047    2551 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 14 11:37:27.625063 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.625061    2551 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 14 11:37:27.625203 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.625079    2551 state_mem.go:36] "Initialized new in-memory state store"
Jul 14 11:37:27.625203 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.625073    2551 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 14 11:37:27.625821 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.625805    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-173.ec2.internal" event="NodeHasSufficientMemory"
Jul 14 11:37:27.625906 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.625834    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-173.ec2.internal" event="NodeHasNoDiskPressure"
Jul 14 11:37:27.625906 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.625849    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-173.ec2.internal" event="NodeHasSufficientPID"
Jul 14 11:37:27.627362 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.627347    2551 policy_none.go:49] "None policy: Start"
Jul 14 11:37:27.627459 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.627366    2551 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 14 11:37:27.627459 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.627379    2551 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 14 11:37:27.627459 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:27.627327    2551 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-173.ec2.internal.18c224f774a2e91f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-173.ec2.internal,UID:ip-10-0-136-173.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-136-173.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-136-173.ec2.internal,},FirstTimestamp:2026-07-14 11:37:27.624493343 +0000 UTC m=+0.300489817,LastTimestamp:2026-07-14 11:37:27.624493343 +0000 UTC m=+0.300489817,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-173.ec2.internal,}"
Jul 14 11:37:27.629464 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.629450    2551 policy_none.go:47] "Start"
Jul 14 11:37:27.633949 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.633687    2551 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-fs9g5"
Jul 14 11:37:27.637538 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:27.637455    2551 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-173.ec2.internal.18c224f774a3329b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-173.ec2.internal,UID:ip-10-0-136-173.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-136-173.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-136-173.ec2.internal,},FirstTimestamp:2026-07-14 11:37:27.624512155 +0000 UTC m=+0.300508629,LastTimestamp:2026-07-14 11:37:27.624512155 +0000 UTC m=+0.300508629,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-173.ec2.internal,}"
Jul 14 11:37:27.646126 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.646104    2551 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-fs9g5"
Jul 14 11:37:27.668089 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.668057    2551 manager.go:337] "Starting Device Plugin manager"
Jul 14 11:37:27.668220 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:27.668104    2551 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 14 11:37:27.668220 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.668117    2551 server.go:87] "Starting device plugin registration server"
Jul 14 11:37:27.668461 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.668447    2551 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 14 11:37:27.668524 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.668465    2551 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 14 11:37:27.668676 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.668603    2551 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 14 11:37:27.668800 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.668761    2551 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 14 11:37:27.668800 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.668775    2551 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 14 11:37:27.669568 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:27.669547    2551 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 14 11:37:27.669639 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:27.669607    2551 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-173.ec2.internal\" not found"
Jul 14 11:37:27.740294 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.740257    2551 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 14 11:37:27.742560 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.740304    2551 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 14 11:37:27.742560 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.740328    2551 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 14 11:37:27.742560 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:27.740362    2551 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 14 11:37:27.746051 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.746033    2551 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 14 11:37:27.768972 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.768940    2551 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 14 11:37:27.770009 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.769991    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-173.ec2.internal" event="NodeHasSufficientMemory"
Jul 14 11:37:27.770106 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.770023    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-173.ec2.internal" event="NodeHasNoDiskPressure"
Jul 14 11:37:27.770106 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.770036    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-173.ec2.internal" event="NodeHasSufficientPID"
Jul 14 11:37:27.770106 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.770060    2551 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-136-173.ec2.internal"
Jul 14 11:37:27.785595 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.785533    2551 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-136-173.ec2.internal"
Jul 14 11:37:27.785595 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:27.785567    2551 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-173.ec2.internal\": node \"ip-10-0-136-173.ec2.internal\" not found"
Jul 14 11:37:27.834706 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:27.834680    2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-173.ec2.internal\" not found"
Jul 14 11:37:27.841028 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.841006    2551 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-173.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-173.ec2.internal"]
Jul 14 11:37:27.841114 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.841073    2551 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 14 11:37:27.842508 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.842489    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-173.ec2.internal" event="NodeHasSufficientMemory"
Jul 14 11:37:27.842625 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.842519    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-173.ec2.internal" event="NodeHasNoDiskPressure"
Jul 14 11:37:27.842625 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.842529    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-173.ec2.internal" event="NodeHasSufficientPID"
Jul 14 11:37:27.844903 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.844880    2551 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 14 11:37:27.845017 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.844998    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-173.ec2.internal"
Jul 14 11:37:27.845061 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.845031    2551 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 14 11:37:27.845617 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.845598    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-173.ec2.internal" event="NodeHasSufficientMemory"
Jul 14 11:37:27.845704 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.845630    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-173.ec2.internal" event="NodeHasNoDiskPressure"
Jul 14 11:37:27.845704 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.845645    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-173.ec2.internal" event="NodeHasSufficientPID"
Jul 14 11:37:27.845704 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.845598    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-173.ec2.internal" event="NodeHasSufficientMemory"
Jul 14 11:37:27.845804 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.845706    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-173.ec2.internal" event="NodeHasNoDiskPressure"
Jul 14 11:37:27.845804 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.845720    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-173.ec2.internal" event="NodeHasSufficientPID"
Jul 14 11:37:27.847971 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.847957    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-173.ec2.internal"
Jul 14 11:37:27.848016 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.847983    2551 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 14 11:37:27.848721 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.848707    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-173.ec2.internal" event="NodeHasSufficientMemory"
Jul 14 11:37:27.848828 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.848736    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-173.ec2.internal" event="NodeHasNoDiskPressure"
Jul 14 11:37:27.848828 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.848751    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-173.ec2.internal" event="NodeHasSufficientPID"
Jul 14 11:37:27.865058 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:27.865031    2551 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-173.ec2.internal\" not found" node="ip-10-0-136-173.ec2.internal"
Jul 14 11:37:27.869652 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:27.869635    2551 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-173.ec2.internal\" not found" node="ip-10-0-136-173.ec2.internal"
Jul 14 11:37:27.904031 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.903995    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6283d946e4456c5cef54b2da21896045-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-173.ec2.internal\" (UID: \"6283d946e4456c5cef54b2da21896045\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-173.ec2.internal"
Jul 14 11:37:27.904031 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.904034    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6283d946e4456c5cef54b2da21896045-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-173.ec2.internal\" (UID: \"6283d946e4456c5cef54b2da21896045\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-173.ec2.internal"
Jul 14 11:37:27.904163 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:27.904052    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a552e96d3a56282f33a24bc1bc8a195b-config\") pod \"kube-apiserver-proxy-ip-10-0-136-173.ec2.internal\" (UID: \"a552e96d3a56282f33a24bc1bc8a195b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-173.ec2.internal"
Jul 14 11:37:27.935705 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:27.935678    2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-173.ec2.internal\" not found"
Jul 14 11:37:28.004543 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.004501    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6283d946e4456c5cef54b2da21896045-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-173.ec2.internal\" (UID: \"6283d946e4456c5cef54b2da21896045\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-173.ec2.internal"
Jul 14 11:37:28.004608 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.004550    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6283d946e4456c5cef54b2da21896045-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-173.ec2.internal\" (UID: \"6283d946e4456c5cef54b2da21896045\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-173.ec2.internal"
Jul 14 11:37:28.004608 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.004569    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a552e96d3a56282f33a24bc1bc8a195b-config\") pod \"kube-apiserver-proxy-ip-10-0-136-173.ec2.internal\" (UID: \"a552e96d3a56282f33a24bc1bc8a195b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-173.ec2.internal"
Jul 14 11:37:28.004672 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.004609    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6283d946e4456c5cef54b2da21896045-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-173.ec2.internal\" (UID: \"6283d946e4456c5cef54b2da21896045\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-173.ec2.internal"
Jul 14 11:37:28.004672 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.004623    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a552e96d3a56282f33a24bc1bc8a195b-config\") pod \"kube-apiserver-proxy-ip-10-0-136-173.ec2.internal\" (UID: \"a552e96d3a56282f33a24bc1bc8a195b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-173.ec2.internal"
Jul 14 11:37:28.004672 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.004628    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6283d946e4456c5cef54b2da21896045-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-173.ec2.internal\" (UID: \"6283d946e4456c5cef54b2da21896045\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-173.ec2.internal"
Jul 14 11:37:28.036104 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:28.036042    2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-173.ec2.internal\" not found"
Jul 14 11:37:28.136889 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:28.136861    2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-173.ec2.internal\" not found"
Jul 14 11:37:28.167058 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.167017    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-173.ec2.internal"
Jul 14 11:37:28.172136 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.171920    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-173.ec2.internal"
Jul 14 11:37:28.236990 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:28.236950    2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-173.ec2.internal\" not found"
Jul 14 11:37:28.337516 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:28.337444    2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-173.ec2.internal\" not found"
Jul 14 11:37:28.437949 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:28.437917    2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-173.ec2.internal\" not found"
Jul 14 11:37:28.465346 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.465315    2551 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 14 11:37:28.502677 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.502637    2551 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-173.ec2.internal"
Jul 14 11:37:28.512259 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.512234    2551 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 14 11:37:28.518145 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.518122    2551 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 14 11:37:28.519476 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.519461    2551 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-173.ec2.internal"
Jul 14 11:37:28.528605 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.528586    2551 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 14 11:37:28.546646 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.546621    2551 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 14 11:37:28.546816 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.546759    2551 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 14 11:37:28.546816 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.546801    2551 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 14 11:37:28.546898 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.546819    2551 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 14 11:37:28.584963 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.584937    2551 apiserver.go:52] "Watching apiserver"
Jul 14 11:37:28.602305 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.602212    2551 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 14 11:37:28.606900 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.606869    2551 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 14 11:37:28.607796 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.607774    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-mldq8","openshift-image-registry/node-ca-27fdx","openshift-multus/multus-ll8fh","openshift-network-operator/iptables-alerter-rpmgk","kube-system/konnectivity-agent-642k8","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qm5ql","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-173.ec2.internal","openshift-multus/multus-additional-cni-plugins-4fhlr","openshift-multus/network-metrics-daemon-66zbn","openshift-network-diagnostics/network-check-target-4zvrn","openshift-ovn-kubernetes/ovnkube-node-cp57b","kube-system/global-pull-secret-syncer-75pwf","kube-system/kube-apiserver-proxy-ip-10-0-136-173.ec2.internal","openshift-cluster-node-tuning-operator/tuned-sjblf"]
Jul 14 11:37:28.610311 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.610288    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mldq8"
Jul 14 11:37:28.612487 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.612461    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-27fdx"
Jul 14 11:37:28.613200 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.613175    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 14 11:37:28.613314 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.613180    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 14 11:37:28.613314 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.613180    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-46h8g\""
Jul 14 11:37:28.614685 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.614665    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ll8fh"
Jul 14 11:37:28.615133 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.615106    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 14 11:37:28.615240 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.615160    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 14 11:37:28.615240 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.615211    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 14 11:37:28.615349 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.615273    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-lxr9q\""
Jul 14 11:37:28.615764 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.615741    2551 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 14 11:37:28.616860 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.616844    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rpmgk"
Jul 14 11:37:28.617092 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.617075    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 14 11:37:28.617186 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.617074    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 14 11:37:28.617460 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.617443    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-ngbcl\""
Jul 14 11:37:28.617546 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.617507    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 14 11:37:28.617605 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.617578    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 14 11:37:28.618951 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.618934    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-642k8"
Jul 14 11:37:28.619289 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.619274    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 14 11:37:28.620166 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.620152    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-wm6ph\""
Jul 14 11:37:28.620893 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.620879    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 14 11:37:28.620951 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.620897    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 14 11:37:28.621150 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.621136    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qm5ql"
Jul 14 11:37:28.621577 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.621559    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-8xk6f\""
Jul 14 11:37:28.621661 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.621602    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 14 11:37:28.621773 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.621759    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 14 11:37:28.623766 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.623510    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4fhlr"
Jul 14 11:37:28.623766 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.623566    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 14 11:37:28.623766 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.623649    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 14 11:37:28.624004 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.623982    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 14 11:37:28.624126 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.624029    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-xtwcs\""
Jul 14 11:37:28.625783 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.625767    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-66zbn"
Jul 14 11:37:28.625870 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:28.625844    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-66zbn" podUID="a5eafaa1-c985-4204-809a-3e9a52c96e55"
Jul 14 11:37:28.626189 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.626173    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-zgzrx\""
Jul 14 11:37:28.626549 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.626534    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 14 11:37:28.626549 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.626547    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 14 11:37:28.628018 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.628002    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4zvrn"
Jul 14 11:37:28.628114 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:28.628083    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4zvrn" podUID="2f44b67b-26da-4ab3-9dfa-343ad6334e4f"
Jul 14 11:37:28.630497 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.630472    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-cp57b"
Jul 14 11:37:28.632770 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.632752    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-75pwf"
Jul 14 11:37:28.632883 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:28.632860    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-75pwf" podUID="3360b04a-23e1-49f9-84e4-11186ccde5e7"
Jul 14 11:37:28.633590 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.633568    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 14 11:37:28.633657 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.633568    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-4wb2t\""
Jul 14 11:37:28.633694 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.633649    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 14 11:37:28.634560 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.634544    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 14 11:37:28.634560 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.634549    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 14 11:37:28.634691 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.634563    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 14 11:37:28.634691 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.634584    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 14 11:37:28.635685 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.635350    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-sjblf"
Jul 14 11:37:28.638248 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.638229    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 14 11:37:28.638332 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.638269    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-4bbp5\""
Jul 14 11:37:28.638579 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.638567    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 14 11:37:28.645487 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.645470    2551 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-v7fdz"
Jul 14 11:37:28.650018 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.649975    2551 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-13 11:32:29 +0000 UTC" deadline="2027-12-17 17:54:40.097690544 +0000 UTC"
Jul 14 11:37:28.650071 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.650018    2551 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12510h17m11.447674757s"
Jul 14 11:37:28.655261 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.655245    2551 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-v7fdz"
Jul 14 11:37:28.704155 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.704138    2551 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 14 11:37:28.706936 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.706913    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1977dd8b-e0ae-4e49-a7d0-ddea4f9a3280-etc-sysctl-d\") pod \"tuned-sjblf\" (UID: \"1977dd8b-e0ae-4e49-a7d0-ddea4f9a3280\") " pod="openshift-cluster-node-tuning-operator/tuned-sjblf"
Jul 14 11:37:28.707053 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.706938    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1977dd8b-e0ae-4e49-a7d0-ddea4f9a3280-etc-sysctl-conf\") pod \"tuned-sjblf\" (UID: \"1977dd8b-e0ae-4e49-a7d0-ddea4f9a3280\") " pod="openshift-cluster-node-tuning-operator/tuned-sjblf"
Jul 14 11:37:28.707053 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.706962    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8a3aafb3-0f38-444f-98c4-dbf37fa6effe-konnectivity-ca\") pod \"konnectivity-agent-642k8\" (UID: \"8a3aafb3-0f38-444f-98c4-dbf37fa6effe\") " pod="kube-system/konnectivity-agent-642k8"
Jul 14 11:37:28.707053 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.706988    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2b7e4c27-d289-460b-b653-a483407a8851-socket-dir\") pod \"aws-ebs-csi-driver-node-qm5ql\" (UID: \"2b7e4c27-d289-460b-b653-a483407a8851\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qm5ql"
Jul 14 11:37:28.707053 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.707003    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2b7e4c27-d289-460b-b653-a483407a8851-device-dir\") pod \"aws-ebs-csi-driver-node-qm5ql\" (UID: \"2b7e4c27-d289-460b-b653-a483407a8851\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qm5ql"
Jul 14 11:37:28.707053 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.707018    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3360b04a-23e1-49f9-84e4-11186ccde5e7-original-pull-secret\") pod \"global-pull-secret-syncer-75pwf\" (UID: \"3360b04a-23e1-49f9-84e4-11186ccde5e7\") " pod="kube-system/global-pull-secret-syncer-75pwf"
Jul 14 11:37:28.707053 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.707034    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/846fd65c-288d-4067-9622-6f652e088853-iptables-alerter-script\") pod \"iptables-alerter-rpmgk\" (UID: \"846fd65c-288d-4067-9622-6f652e088853\") " pod="openshift-network-operator/iptables-alerter-rpmgk"
Jul 14 11:37:28.707266 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.707056    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8a3aafb3-0f38-444f-98c4-dbf37fa6effe-agent-certs\") pod \"konnectivity-agent-642k8\" (UID: \"8a3aafb3-0f38-444f-98c4-dbf37fa6effe\") " pod="kube-system/konnectivity-agent-642k8"
Jul 14 11:37:28.707266 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.707100    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e356b532-03eb-4760-93eb-a731dbc0f142-tmp-dir\") pod \"node-resolver-mldq8\" (UID: \"e356b532-03eb-4760-93eb-a731dbc0f142\") " pod="openshift-dns/node-resolver-mldq8"
Jul 14 11:37:28.707266 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.707153    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/62c98a11-ece4-4497-99e7-0840a7626a56-host-slash\") pod \"ovnkube-node-cp57b\" (UID: \"62c98a11-ece4-4497-99e7-0840a7626a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cp57b"
Jul 14 11:37:28.707266 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.707180    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77qmm\" (UniqueName: \"kubernetes.io/projected/2f44b67b-26da-4ab3-9dfa-343ad6334e4f-kube-api-access-77qmm\") pod \"network-check-target-4zvrn\" (UID: \"2f44b67b-26da-4ab3-9dfa-343ad6334e4f\") " pod="openshift-network-diagnostics/network-check-target-4zvrn"
Jul 14 11:37:28.707266 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.707199    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3bb65b10-6f3c-43ee-8ad9-060bfb4df18e-cni-binary-copy\") pod \"multus-ll8fh\" (UID: \"3bb65b10-6f3c-43ee-8ad9-060bfb4df18e\") " pod="openshift-multus/multus-ll8fh"
Jul 14 11:37:28.707266 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.707225    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3bb65b10-6f3c-43ee-8ad9-060bfb4df18e-hostroot\") pod \"multus-ll8fh\" (UID: \"3bb65b10-6f3c-43ee-8ad9-060bfb4df18e\") " pod="openshift-multus/multus-ll8fh"
Jul 14 11:37:28.707266 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.707247    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4c7434bb-d4dd-407e-a009-33d672d4bfc6-cnibin\") pod \"multus-additional-cni-plugins-4fhlr\" (UID: \"4c7434bb-d4dd-407e-a009-33d672d4bfc6\") " pod="openshift-multus/multus-additional-cni-plugins-4fhlr"
Jul 14 11:37:28.707266 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.707262    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4c7434bb-d4dd-407e-a009-33d672d4bfc6-os-release\") pod \"multus-additional-cni-plugins-4fhlr\" (UID: \"4c7434bb-d4dd-407e-a009-33d672d4bfc6\") " pod="openshift-multus/multus-additional-cni-plugins-4fhlr"
Jul 14 11:37:28.707527 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.707287    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/62c98a11-ece4-4497-99e7-0840a7626a56-host-kubelet\") pod \"ovnkube-node-cp57b\" (UID: \"62c98a11-ece4-4497-99e7-0840a7626a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cp57b"
Jul 14 11:37:28.707527 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.707320    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/62c98a11-ece4-4497-99e7-0840a7626a56-etc-openvswitch\") pod \"ovnkube-node-cp57b\" (UID: \"62c98a11-ece4-4497-99e7-0840a7626a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cp57b"
Jul 14 11:37:28.707527 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.707339    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1977dd8b-e0ae-4e49-a7d0-ddea4f9a3280-etc-kubernetes\") pod \"tuned-sjblf\" (UID: \"1977dd8b-e0ae-4e49-a7d0-ddea4f9a3280\") " pod="openshift-cluster-node-tuning-operator/tuned-sjblf"
Jul 14 11:37:28.707527 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.707356    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3bb65b10-6f3c-43ee-8ad9-060bfb4df18e-cnibin\") pod \"multus-ll8fh\" (UID: \"3bb65b10-6f3c-43ee-8ad9-060bfb4df18e\") " pod="openshift-multus/multus-ll8fh"
Jul 14 11:37:28.707527 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.707370    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3bb65b10-6f3c-43ee-8ad9-060bfb4df18e-multus-conf-dir\") pod \"multus-ll8fh\" (UID: \"3bb65b10-6f3c-43ee-8ad9-060bfb4df18e\") " pod="openshift-multus/multus-ll8fh"
Jul 14 11:37:28.707527 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.707383    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/62c98a11-ece4-4497-99e7-0840a7626a56-host-cni-bin\") pod \"ovnkube-node-cp57b\" (UID: \"62c98a11-ece4-4497-99e7-0840a7626a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cp57b"
Jul 14 11:37:28.707527 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.707420    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1977dd8b-e0ae-4e49-a7d0-ddea4f9a3280-var-lib-kubelet\") pod \"tuned-sjblf\" (UID: \"1977dd8b-e0ae-4e49-a7d0-ddea4f9a3280\") " pod="openshift-cluster-node-tuning-operator/tuned-sjblf"
Jul 14 11:37:28.707527 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.707440    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1977dd8b-e0ae-4e49-a7d0-ddea4f9a3280-host\") pod \"tuned-sjblf\" (UID: \"1977dd8b-e0ae-4e49-a7d0-ddea4f9a3280\") " pod="openshift-cluster-node-tuning-operator/tuned-sjblf"
Jul 14 11:37:28.707527 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.707460    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmjhv\" (UniqueName: \"kubernetes.io/projected/2b7e4c27-d289-460b-b653-a483407a8851-kube-api-access-wmjhv\") pod \"aws-ebs-csi-driver-node-qm5ql\" (UID: \"2b7e4c27-d289-460b-b653-a483407a8851\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qm5ql"
Jul 14 11:37:28.707527 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.707495    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3bb65b10-6f3c-43ee-8ad9-060bfb4df18e-host-var-lib-cni-multus\") pod \"multus-ll8fh\" (UID: \"3bb65b10-6f3c-43ee-8ad9-060bfb4df18e\") " pod="openshift-multus/multus-ll8fh"
Jul 14 11:37:28.707527 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.707514    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4c7434bb-d4dd-407e-a009-33d672d4bfc6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4fhlr\" (UID: \"4c7434bb-d4dd-407e-a009-33d672d4bfc6\") " pod="openshift-multus/multus-additional-cni-plugins-4fhlr"
Jul 14 11:37:28.707527 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.707530    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/846fd65c-288d-4067-9622-6f652e088853-host-slash\") pod \"iptables-alerter-rpmgk\" (UID: \"846fd65c-288d-4067-9622-6f652e088853\") " pod="openshift-network-operator/iptables-alerter-rpmgk"
Jul 14 11:37:28.707854 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.707545    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5eafaa1-c985-4204-809a-3e9a52c96e55-metrics-certs\") pod \"network-metrics-daemon-66zbn\" (UID: \"a5eafaa1-c985-4204-809a-3e9a52c96e55\") " pod="openshift-multus/network-metrics-daemon-66zbn"
Jul 14 11:37:28.707854 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.707577    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/62c98a11-ece4-4497-99e7-0840a7626a56-host-run-ovn-kubernetes\") pod \"ovnkube-node-cp57b\" (UID: \"62c98a11-ece4-4497-99e7-0840a7626a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cp57b"
Jul 14 11:37:28.707854 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.707599    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2b7e4c27-d289-460b-b653-a483407a8851-registration-dir\") pod \"aws-ebs-csi-driver-node-qm5ql\" (UID: \"2b7e4c27-d289-460b-b653-a483407a8851\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qm5ql"
Jul 14 11:37:28.707854 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.707613    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2b7e4c27-d289-460b-b653-a483407a8851-etc-selinux\") pod \"aws-ebs-csi-driver-node-qm5ql\" (UID: \"2b7e4c27-d289-460b-b653-a483407a8851\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qm5ql"
Jul 14 11:37:28.707854 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.707627    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3bb65b10-6f3c-43ee-8ad9-060bfb4df18e-host-run-multus-certs\") pod \"multus-ll8fh\" (UID: \"3bb65b10-6f3c-43ee-8ad9-060bfb4df18e\") " pod="openshift-multus/multus-ll8fh"
Jul 14 11:37:28.707854 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.707641    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/62c98a11-ece4-4497-99e7-0840a7626a56-run-openvswitch\") pod \"ovnkube-node-cp57b\" (UID: \"62c98a11-ece4-4497-99e7-0840a7626a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cp57b"
Jul 14 11:37:28.707854 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.707663    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1977dd8b-e0ae-4e49-a7d0-ddea4f9a3280-etc-modprobe-d\") pod \"tuned-sjblf\" (UID: \"1977dd8b-e0ae-4e49-a7d0-ddea4f9a3280\") " pod="openshift-cluster-node-tuning-operator/tuned-sjblf"
Jul 14 11:37:28.707854 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.707679    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1977dd8b-e0ae-4e49-a7d0-ddea4f9a3280-etc-sysconfig\") pod \"tuned-sjblf\" (UID: \"1977dd8b-e0ae-4e49-a7d0-ddea4f9a3280\") " pod="openshift-cluster-node-tuning-operator/tuned-sjblf"
Jul 14 11:37:28.707854 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.707700    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1977dd8b-e0ae-4e49-a7d0-ddea4f9a3280-sys\") pod \"tuned-sjblf\" (UID: \"1977dd8b-e0ae-4e49-a7d0-ddea4f9a3280\") " pod="openshift-cluster-node-tuning-operator/tuned-sjblf"
Jul 14 11:37:28.707854 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.707713    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1977dd8b-e0ae-4e49-a7d0-ddea4f9a3280-tmp\") pod \"tuned-sjblf\" (UID: \"1977dd8b-e0ae-4e49-a7d0-ddea4f9a3280\") " pod="openshift-cluster-node-tuning-operator/tuned-sjblf"
Jul 14 11:37:28.707854 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.707734    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3bb65b10-6f3c-43ee-8ad9-060bfb4df18e-host-var-lib-kubelet\") pod \"multus-ll8fh\" (UID: \"3bb65b10-6f3c-43ee-8ad9-060bfb4df18e\") " pod="openshift-multus/multus-ll8fh"
Jul 14 11:37:28.707854 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.707766    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4c7434bb-d4dd-407e-a009-33d672d4bfc6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4fhlr\" (UID: \"4c7434bb-d4dd-407e-a009-33d672d4bfc6\") " pod="openshift-multus/multus-additional-cni-plugins-4fhlr"
Jul 14 11:37:28.707854 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.707784    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mdh8\" (UniqueName: \"kubernetes.io/projected/4c7434bb-d4dd-407e-a009-33d672d4bfc6-kube-api-access-7mdh8\") pod \"multus-additional-cni-plugins-4fhlr\" (UID: \"4c7434bb-d4dd-407e-a009-33d672d4bfc6\") " pod="openshift-multus/multus-additional-cni-plugins-4fhlr"
Jul 14 11:37:28.707854 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.707800    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/62c98a11-ece4-4497-99e7-0840a7626a56-run-ovn\") pod \"ovnkube-node-cp57b\" (UID: \"62c98a11-ece4-4497-99e7-0840a7626a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cp57b"
Jul 14 11:37:28.707854 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.707823    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/62c98a11-ece4-4497-99e7-0840a7626a56-log-socket\") pod \"ovnkube-node-cp57b\" (UID: \"62c98a11-ece4-4497-99e7-0840a7626a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cp57b"
Jul 14 11:37:28.707854 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.707837    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gnnl\" (UniqueName: \"kubernetes.io/projected/1977dd8b-e0ae-4e49-a7d0-ddea4f9a3280-kube-api-access-5gnnl\") pod \"tuned-sjblf\" (UID: \"1977dd8b-e0ae-4e49-a7d0-ddea4f9a3280\") " pod="openshift-cluster-node-tuning-operator/tuned-sjblf"
Jul 14 11:37:28.708277 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.707874    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/62c98a11-ece4-4497-99e7-0840a7626a56-host-cni-netd\") pod \"ovnkube-node-cp57b\" (UID: \"62c98a11-ece4-4497-99e7-0840a7626a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cp57b"
Jul 14 11:37:28.708277 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.707896    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3bb65b10-6f3c-43ee-8ad9-060bfb4df18e-system-cni-dir\") pod \"multus-ll8fh\" (UID: \"3bb65b10-6f3c-43ee-8ad9-060bfb4df18e\") " pod="openshift-multus/multus-ll8fh"
Jul 14 11:37:28.708277 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.707910    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3bb65b10-6f3c-43ee-8ad9-060bfb4df18e-multus-socket-dir-parent\") pod \"multus-ll8fh\" (UID: \"3bb65b10-6f3c-43ee-8ad9-060bfb4df18e\") " pod="openshift-multus/multus-ll8fh"
Jul 14 11:37:28.708277 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.707930    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/62c98a11-ece4-4497-99e7-0840a7626a56-run-systemd\") pod \"ovnkube-node-cp57b\" (UID: \"62c98a11-ece4-4497-99e7-0840a7626a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cp57b"
Jul 14 11:37:28.708277 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.707943    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/62c98a11-ece4-4497-99e7-0840a7626a56-node-log\") pod \"ovnkube-node-cp57b\" (UID: \"62c98a11-ece4-4497-99e7-0840a7626a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cp57b"
Jul 14 11:37:28.708277 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.707966    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/62c98a11-ece4-4497-99e7-0840a7626a56-ovnkube-config\") pod \"ovnkube-node-cp57b\" (UID: \"62c98a11-ece4-4497-99e7-0840a7626a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cp57b"
Jul 14 11:37:28.708277 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.708001    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1977dd8b-e0ae-4e49-a7d0-ddea4f9a3280-lib-modules\") pod \"tuned-sjblf\" (UID: \"1977dd8b-e0ae-4e49-a7d0-ddea4f9a3280\") " pod="openshift-cluster-node-tuning-operator/tuned-sjblf"
Jul 14 11:37:28.708277 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.708019    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2b7e4c27-d289-460b-b653-a483407a8851-sys-fs\") pod \"aws-ebs-csi-driver-node-qm5ql\" (UID: \"2b7e4c27-d289-460b-b653-a483407a8851\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qm5ql"
Jul 14 11:37:28.708277 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.708032    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e356b532-03eb-4760-93eb-a731dbc0f142-hosts-file\") pod \"node-resolver-mldq8\" (UID: \"e356b532-03eb-4760-93eb-a731dbc0f142\") " pod="openshift-dns/node-resolver-mldq8"
Jul 14 11:37:28.708277 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.708047    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1977dd8b-e0ae-4e49-a7d0-ddea4f9a3280-run\") pod \"tuned-sjblf\" (UID: \"1977dd8b-e0ae-4e49-a7d0-ddea4f9a3280\") " pod="openshift-cluster-node-tuning-operator/tuned-sjblf"
Jul 14 11:37:28.708277 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.708066    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3bb65b10-6f3c-43ee-8ad9-060bfb4df18e-multus-cni-dir\") pod \"multus-ll8fh\" (UID: \"3bb65b10-6f3c-43ee-8ad9-060bfb4df18e\") " pod="openshift-multus/multus-ll8fh"
Jul 14 11:37:28.708277 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.708089    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3bb65b10-6f3c-43ee-8ad9-060bfb4df18e-host-var-lib-cni-bin\") pod \"multus-ll8fh\" (UID: \"3bb65b10-6f3c-43ee-8ad9-060bfb4df18e\") " pod="openshift-multus/multus-ll8fh"
Jul 14 11:37:28.708277 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.708107    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4c7434bb-d4dd-407e-a009-33d672d4bfc6-system-cni-dir\") pod \"multus-additional-cni-plugins-4fhlr\" (UID: \"4c7434bb-d4dd-407e-a009-33d672d4bfc6\") " pod="openshift-multus/multus-additional-cni-plugins-4fhlr"
Jul 14 11:37:28.708277 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.708131    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4c7434bb-d4dd-407e-a009-33d672d4bfc6-cni-binary-copy\") pod \"multus-additional-cni-plugins-4fhlr\" (UID: \"4c7434bb-d4dd-407e-a009-33d672d4bfc6\") " pod="openshift-multus/multus-additional-cni-plugins-4fhlr"
Jul 14 11:37:28.708277 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.708147    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/62c98a11-ece4-4497-99e7-0840a7626a56-systemd-units\") pod \"ovnkube-node-cp57b\" (UID: \"62c98a11-ece4-4497-99e7-0840a7626a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cp57b"
Jul 14 11:37:28.708277 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.708163    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/62c98a11-ece4-4497-99e7-0840a7626a56-var-lib-openvswitch\") pod \"ovnkube-node-cp57b\" (UID: \"62c98a11-ece4-4497-99e7-0840a7626a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cp57b"
Jul 14 11:37:28.708277 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.708188    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/62c98a11-ece4-4497-99e7-0840a7626a56-env-overrides\") pod \"ovnkube-node-cp57b\" (UID: \"62c98a11-ece4-4497-99e7-0840a7626a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cp57b"
Jul 14 11:37:28.708813 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.708215    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3360b04a-23e1-49f9-84e4-11186ccde5e7-kubelet-config\") pod \"global-pull-secret-syncer-75pwf\" (UID: \"3360b04a-23e1-49f9-84e4-11186ccde5e7\") " pod="kube-system/global-pull-secret-syncer-75pwf"
Jul 14 11:37:28.708813 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.708256    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/62c98a11-ece4-4497-99e7-0840a7626a56-ovn-node-metrics-cert\") pod \"ovnkube-node-cp57b\" (UID: \"62c98a11-ece4-4497-99e7-0840a7626a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cp57b"
Jul 14 11:37:28.708813 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.708278    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k86gd\" (UniqueName: \"kubernetes.io/projected/a5eafaa1-c985-4204-809a-3e9a52c96e55-kube-api-access-k86gd\") pod \"network-metrics-daemon-66zbn\" (UID: \"a5eafaa1-c985-4204-809a-3e9a52c96e55\") " pod="openshift-multus/network-metrics-daemon-66zbn"
Jul 14 11:37:28.708813 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.708304    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffnvs\" (UniqueName: \"kubernetes.io/projected/e356b532-03eb-4760-93eb-a731dbc0f142-kube-api-access-ffnvs\") pod \"node-resolver-mldq8\" (UID: \"e356b532-03eb-4760-93eb-a731dbc0f142\") " pod="openshift-dns/node-resolver-mldq8"
Jul 14 11:37:28.708813 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.708325    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8590baa3-db08-49ef-8d26-3f551d6954e4-serviceca\") pod \"node-ca-27fdx\" (UID: \"8590baa3-db08-49ef-8d26-3f551d6954e4\") " pod="openshift-image-registry/node-ca-27fdx"
Jul 14 11:37:28.708813 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.708340    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdhqq\" (UniqueName: \"kubernetes.io/projected/8590baa3-db08-49ef-8d26-3f551d6954e4-kube-api-access-jdhqq\") pod \"node-ca-27fdx\" (UID: \"8590baa3-db08-49ef-8d26-3f551d6954e4\") " pod="openshift-image-registry/node-ca-27fdx"
Jul 14 11:37:28.708813 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.708354    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3bb65b10-6f3c-43ee-8ad9-060bfb4df18e-host-run-netns\") pod \"multus-ll8fh\" (UID: \"3bb65b10-6f3c-43ee-8ad9-060bfb4df18e\") " pod="openshift-multus/multus-ll8fh"
Jul 14 11:37:28.708813 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.708369    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3bb65b10-6f3c-43ee-8ad9-060bfb4df18e-multus-daemon-config\") pod \"multus-ll8fh\" (UID: \"3bb65b10-6f3c-43ee-8ad9-060bfb4df18e\") " pod="openshift-multus/multus-ll8fh"
Jul 14 11:37:28.708813 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.708406    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65mtw\" (UniqueName: \"kubernetes.io/projected/3bb65b10-6f3c-43ee-8ad9-060bfb4df18e-kube-api-access-65mtw\") pod \"multus-ll8fh\" (UID: \"3bb65b10-6f3c-43ee-8ad9-060bfb4df18e\") " pod="openshift-multus/multus-ll8fh"
Jul 14 11:37:28.708813 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.708427    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/62c98a11-ece4-4497-99e7-0840a7626a56-ovnkube-script-lib\") pod \"ovnkube-node-cp57b\" (UID: \"62c98a11-ece4-4497-99e7-0840a7626a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cp57b"
Jul 14 11:37:28.708813 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.708449    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3bb65b10-6f3c-43ee-8ad9-060bfb4df18e-os-release\") pod \"multus-ll8fh\" (UID: \"3bb65b10-6f3c-43ee-8ad9-060bfb4df18e\") " pod="openshift-multus/multus-ll8fh"
Jul 14 11:37:28.708813 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.708481    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3bb65b10-6f3c-43ee-8ad9-060bfb4df18e-host-run-k8s-cni-cncf-io\") pod \"multus-ll8fh\" (UID: \"3bb65b10-6f3c-43ee-8ad9-060bfb4df18e\") " pod="openshift-multus/multus-ll8fh"
Jul 14 11:37:28.708813 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.708502    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4c7434bb-d4dd-407e-a009-33d672d4bfc6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4fhlr\" (UID: \"4c7434bb-d4dd-407e-a009-33d672d4bfc6\") " pod="openshift-multus/multus-additional-cni-plugins-4fhlr"
Jul 14 11:37:28.708813 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.708525    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/62c98a11-ece4-4497-99e7-0840a7626a56-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cp57b\" (UID: \"62c98a11-ece4-4497-99e7-0840a7626a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cp57b"
Jul 14 11:37:28.708813 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.708548    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pqmx\" (UniqueName: \"kubernetes.io/projected/62c98a11-ece4-4497-99e7-0840a7626a56-kube-api-access-5pqmx\") pod \"ovnkube-node-cp57b\" (UID: \"62c98a11-ece4-4497-99e7-0840a7626a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cp57b"
Jul 14 11:37:28.708813 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.708567    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3360b04a-23e1-49f9-84e4-11186ccde5e7-dbus\") pod \"global-pull-secret-syncer-75pwf\" (UID: \"3360b04a-23e1-49f9-84e4-11186ccde5e7\") " pod="kube-system/global-pull-secret-syncer-75pwf"
Jul 14 11:37:28.709533 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.708586    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8590baa3-db08-49ef-8d26-3f551d6954e4-host\") pod \"node-ca-27fdx\" (UID: \"8590baa3-db08-49ef-8d26-3f551d6954e4\") " pod="openshift-image-registry/node-ca-27fdx"
Jul 14 11:37:28.709533 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.708599    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3bb65b10-6f3c-43ee-8ad9-060bfb4df18e-etc-kubernetes\") pod \"multus-ll8fh\" (UID: \"3bb65b10-6f3c-43ee-8ad9-060bfb4df18e\") " pod="openshift-multus/multus-ll8fh"
Jul 14 11:37:28.709533 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.708619    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/62c98a11-ece4-4497-99e7-0840a7626a56-host-run-netns\") pod \"ovnkube-node-cp57b\" (UID: \"62c98a11-ece4-4497-99e7-0840a7626a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cp57b"
Jul 14 11:37:28.709533 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.708654    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1977dd8b-e0ae-4e49-a7d0-ddea4f9a3280-etc-systemd\") pod \"tuned-sjblf\" (UID: \"1977dd8b-e0ae-4e49-a7d0-ddea4f9a3280\") " pod="openshift-cluster-node-tuning-operator/tuned-sjblf"
Jul 14 11:37:28.709533 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.708669    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1977dd8b-e0ae-4e49-a7d0-ddea4f9a3280-etc-tuned\") pod \"tuned-sjblf\" (UID: \"1977dd8b-e0ae-4e49-a7d0-ddea4f9a3280\") " pod="openshift-cluster-node-tuning-operator/tuned-sjblf"
Jul 14 11:37:28.709533 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.708685    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9lbt\" (UniqueName: \"kubernetes.io/projected/846fd65c-288d-4067-9622-6f652e088853-kube-api-access-z9lbt\") pod \"iptables-alerter-rpmgk\" (UID: \"846fd65c-288d-4067-9622-6f652e088853\") " pod="openshift-network-operator/iptables-alerter-rpmgk"
Jul 14 11:37:28.709533 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.708699    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2b7e4c27-d289-460b-b653-a483407a8851-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qm5ql\" (UID: \"2b7e4c27-d289-460b-b653-a483407a8851\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qm5ql"
Jul 14 11:37:28.715595 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:28.715569    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6283d946e4456c5cef54b2da21896045.slice/crio-0fafa8b8e8970bc9927dd1ff919ffe9e3b88de21d795ea64cad24a8c0dedbdfe WatchSource:0}: Error finding container 0fafa8b8e8970bc9927dd1ff919ffe9e3b88de21d795ea64cad24a8c0dedbdfe: Status 404 returned error can't find the container with id 0fafa8b8e8970bc9927dd1ff919ffe9e3b88de21d795ea64cad24a8c0dedbdfe
Jul 14 11:37:28.715832 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:28.715820    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda552e96d3a56282f33a24bc1bc8a195b.slice/crio-f7d3728c3b5f95db59ef7d7fd537cb995156a5ffddb9baa1cfd58f051f0746d1 WatchSource:0}: Error finding container f7d3728c3b5f95db59ef7d7fd537cb995156a5ffddb9baa1cfd58f051f0746d1: Status 404 returned error can't find the container with id f7d3728c3b5f95db59ef7d7fd537cb995156a5ffddb9baa1cfd58f051f0746d1
Jul 14 11:37:28.720253 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.720232    2551 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 14 11:37:28.743223 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.743174    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-173.ec2.internal" event={"ID":"6283d946e4456c5cef54b2da21896045","Type":"ContainerStarted","Data":"0fafa8b8e8970bc9927dd1ff919ffe9e3b88de21d795ea64cad24a8c0dedbdfe"}
Jul 14 11:37:28.744284 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.744261    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-173.ec2.internal" event={"ID":"a552e96d3a56282f33a24bc1bc8a195b","Type":"ContainerStarted","Data":"f7d3728c3b5f95db59ef7d7fd537cb995156a5ffddb9baa1cfd58f051f0746d1"}
Jul 14 11:37:28.809486 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.809447    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/62c98a11-ece4-4497-99e7-0840a7626a56-host-cni-netd\") pod \"ovnkube-node-cp57b\" (UID: \"62c98a11-ece4-4497-99e7-0840a7626a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cp57b"
Jul 14 11:37:28.809486 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.809482    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3bb65b10-6f3c-43ee-8ad9-060bfb4df18e-system-cni-dir\") pod \"multus-ll8fh\" (UID: \"3bb65b10-6f3c-43ee-8ad9-060bfb4df18e\") " pod="openshift-multus/multus-ll8fh"
Jul 14 11:37:28.809486 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.809499    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3bb65b10-6f3c-43ee-8ad9-060bfb4df18e-multus-socket-dir-parent\") pod \"multus-ll8fh\" (UID: \"3bb65b10-6f3c-43ee-8ad9-060bfb4df18e\") " pod="openshift-multus/multus-ll8fh"
Jul 14 11:37:28.809796 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.809516    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/62c98a11-ece4-4497-99e7-0840a7626a56-run-systemd\") pod \"ovnkube-node-cp57b\" (UID: \"62c98a11-ece4-4497-99e7-0840a7626a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cp57b"
Jul 14 11:37:28.809796 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.809531    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/62c98a11-ece4-4497-99e7-0840a7626a56-node-log\") pod \"ovnkube-node-cp57b\" (UID: \"62c98a11-ece4-4497-99e7-0840a7626a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cp57b"
Jul 14 11:37:28.809796 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.809548    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/62c98a11-ece4-4497-99e7-0840a7626a56-ovnkube-config\") pod \"ovnkube-node-cp57b\" (UID: \"62c98a11-ece4-4497-99e7-0840a7626a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cp57b"
Jul 14 11:37:28.809796 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.809561    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1977dd8b-e0ae-4e49-a7d0-ddea4f9a3280-lib-modules\") pod \"tuned-sjblf\" (UID: \"1977dd8b-e0ae-4e49-a7d0-ddea4f9a3280\") " pod="openshift-cluster-node-tuning-operator/tuned-sjblf"
Jul 14 11:37:28.809796 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.809589    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/62c98a11-ece4-4497-99e7-0840a7626a56-run-systemd\") pod \"ovnkube-node-cp57b\" (UID: \"62c98a11-ece4-4497-99e7-0840a7626a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cp57b"
Jul 14 11:37:28.809796 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.809600    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/62c98a11-ece4-4497-99e7-0840a7626a56-node-log\") pod \"ovnkube-node-cp57b\" (UID: \"62c98a11-ece4-4497-99e7-0840a7626a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cp57b"
Jul 14 11:37:28.809796 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.809607    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3bb65b10-6f3c-43ee-8ad9-060bfb4df18e-multus-socket-dir-parent\") pod \"multus-ll8fh\" (UID: \"3bb65b10-6f3c-43ee-8ad9-060bfb4df18e\") " pod="openshift-multus/multus-ll8fh"
Jul 14 11:37:28.809796 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.809589    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/62c98a11-ece4-4497-99e7-0840a7626a56-host-cni-netd\") pod \"ovnkube-node-cp57b\" (UID: \"62c98a11-ece4-4497-99e7-0840a7626a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cp57b"
Jul 14 11:37:28.809796 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.809601    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2b7e4c27-d289-460b-b653-a483407a8851-sys-fs\") pod \"aws-ebs-csi-driver-node-qm5ql\" (UID: \"2b7e4c27-d289-460b-b653-a483407a8851\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qm5ql"
Jul 14 11:37:28.809796 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.809655    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3bb65b10-6f3c-43ee-8ad9-060bfb4df18e-system-cni-dir\") pod \"multus-ll8fh\" (UID: \"3bb65b10-6f3c-43ee-8ad9-060bfb4df18e\") " pod="openshift-multus/multus-ll8fh"
Jul 14 11:37:28.809796 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.809667    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2b7e4c27-d289-460b-b653-a483407a8851-sys-fs\") pod \"aws-ebs-csi-driver-node-qm5ql\" (UID: \"2b7e4c27-d289-460b-b653-a483407a8851\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qm5ql"
Jul 14 11:37:28.809796 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.809661    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e356b532-03eb-4760-93eb-a731dbc0f142-hosts-file\") pod \"node-resolver-mldq8\" (UID: \"e356b532-03eb-4760-93eb-a731dbc0f142\") " pod="openshift-dns/node-resolver-mldq8"
Jul 14 11:37:28.809796 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.809697    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e356b532-03eb-4760-93eb-a731dbc0f142-hosts-file\") pod \"node-resolver-mldq8\" (UID: \"e356b532-03eb-4760-93eb-a731dbc0f142\") " pod="openshift-dns/node-resolver-mldq8"
Jul 14 11:37:28.809796 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.809706    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1977dd8b-e0ae-4e49-a7d0-ddea4f9a3280-run\") pod \"tuned-sjblf\" (UID: \"1977dd8b-e0ae-4e49-a7d0-ddea4f9a3280\") " pod="openshift-cluster-node-tuning-operator/tuned-sjblf"
Jul 14 11:37:28.809796 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.809719    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1977dd8b-e0ae-4e49-a7d0-ddea4f9a3280-lib-modules\") pod \"tuned-sjblf\" (UID: \"1977dd8b-e0ae-4e49-a7d0-ddea4f9a3280\") " pod="openshift-cluster-node-tuning-operator/tuned-sjblf"
Jul 14 11:37:28.809796 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.809728    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3bb65b10-6f3c-43ee-8ad9-060bfb4df18e-multus-cni-dir\") pod \"multus-ll8fh\" (UID: \"3bb65b10-6f3c-43ee-8ad9-060bfb4df18e\") " pod="openshift-multus/multus-ll8fh"
Jul 14 11:37:28.809796 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.809747    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1977dd8b-e0ae-4e49-a7d0-ddea4f9a3280-run\") pod \"tuned-sjblf\" (UID: \"1977dd8b-e0ae-4e49-a7d0-ddea4f9a3280\") " pod="openshift-cluster-node-tuning-operator/tuned-sjblf"
Jul 14 11:37:28.809796 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.809748    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3bb65b10-6f3c-43ee-8ad9-060bfb4df18e-host-var-lib-cni-bin\") pod \"multus-ll8fh\" (UID: \"3bb65b10-6f3c-43ee-8ad9-060bfb4df18e\") " pod="openshift-multus/multus-ll8fh"
Jul 14 11:37:28.810576 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.809780    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3bb65b10-6f3c-43ee-8ad9-060bfb4df18e-host-var-lib-cni-bin\") pod \"multus-ll8fh\" (UID: \"3bb65b10-6f3c-43ee-8ad9-060bfb4df18e\") " pod="openshift-multus/multus-ll8fh"
Jul 14 11:37:28.810576 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.809790    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4c7434bb-d4dd-407e-a009-33d672d4bfc6-system-cni-dir\") pod \"multus-additional-cni-plugins-4fhlr\" (UID: \"4c7434bb-d4dd-407e-a009-33d672d4bfc6\") " pod="openshift-multus/multus-additional-cni-plugins-4fhlr"
Jul 14 11:37:28.810576 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.809823    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4c7434bb-d4dd-407e-a009-33d672d4bfc6-cni-binary-copy\") pod \"multus-additional-cni-plugins-4fhlr\" (UID: \"4c7434bb-d4dd-407e-a009-33d672d4bfc6\") " pod="openshift-multus/multus-additional-cni-plugins-4fhlr"
Jul 14 11:37:28.810576 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.809842    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/62c98a11-ece4-4497-99e7-0840a7626a56-systemd-units\") pod \"ovnkube-node-cp57b\" (UID: \"62c98a11-ece4-4497-99e7-0840a7626a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cp57b"
Jul 14 11:37:28.810576 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.809858    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/62c98a11-ece4-4497-99e7-0840a7626a56-var-lib-openvswitch\") pod \"ovnkube-node-cp57b\" (UID: \"62c98a11-ece4-4497-99e7-0840a7626a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cp57b"
Jul 14 11:37:28.810576 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.809822    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4c7434bb-d4dd-407e-a009-33d672d4bfc6-system-cni-dir\") pod \"multus-additional-cni-plugins-4fhlr\" (UID: \"4c7434bb-d4dd-407e-a009-33d672d4bfc6\") " pod="openshift-multus/multus-additional-cni-plugins-4fhlr"
Jul 14 11:37:28.810576 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.809887    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3bb65b10-6f3c-43ee-8ad9-060bfb4df18e-multus-cni-dir\") pod \"multus-ll8fh\" (UID: \"3bb65b10-6f3c-43ee-8ad9-060bfb4df18e\") " pod="openshift-multus/multus-ll8fh"
Jul 14 11:37:28.810576 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.809875    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/62c98a11-ece4-4497-99e7-0840a7626a56-env-overrides\") pod \"ovnkube-node-cp57b\" (UID: \"62c98a11-ece4-4497-99e7-0840a7626a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cp57b"
Jul 14 11:37:28.810576 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.809936    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/62c98a11-ece4-4497-99e7-0840a7626a56-var-lib-openvswitch\") pod \"ovnkube-node-cp57b\" (UID: \"62c98a11-ece4-4497-99e7-0840a7626a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cp57b"
Jul 14 11:37:28.810576 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.809996    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3360b04a-23e1-49f9-84e4-11186ccde5e7-kubelet-config\") pod \"global-pull-secret-syncer-75pwf\" (UID: \"3360b04a-23e1-49f9-84e4-11186ccde5e7\") " pod="kube-system/global-pull-secret-syncer-75pwf"
Jul 14 11:37:28.810576 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.810023    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/62c98a11-ece4-4497-99e7-0840a7626a56-ovn-node-metrics-cert\") pod \"ovnkube-node-cp57b\" (UID: \"62c98a11-ece4-4497-99e7-0840a7626a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cp57b"
Jul 14 11:37:28.810576 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.810049    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k86gd\" (UniqueName: \"kubernetes.io/projected/a5eafaa1-c985-4204-809a-3e9a52c96e55-kube-api-access-k86gd\") pod \"network-metrics-daemon-66zbn\" (UID: \"a5eafaa1-c985-4204-809a-3e9a52c96e55\") " pod="openshift-multus/network-metrics-daemon-66zbn"
Jul 14 11:37:28.810576 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.810052    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/62c98a11-ece4-4497-99e7-0840a7626a56-systemd-units\") pod \"ovnkube-node-cp57b\" (UID: \"62c98a11-ece4-4497-99e7-0840a7626a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cp57b"
Jul 14 11:37:28.810576 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.810098    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3360b04a-23e1-49f9-84e4-11186ccde5e7-kubelet-config\") pod \"global-pull-secret-syncer-75pwf\" (UID: \"3360b04a-23e1-49f9-84e4-11186ccde5e7\") " pod="kube-system/global-pull-secret-syncer-75pwf"
Jul 14 11:37:28.810576 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.810075    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffnvs\" (UniqueName: \"kubernetes.io/projected/e356b532-03eb-4760-93eb-a731dbc0f142-kube-api-access-ffnvs\") pod \"node-resolver-mldq8\" (UID: \"e356b532-03eb-4760-93eb-a731dbc0f142\") " pod="openshift-dns/node-resolver-mldq8"
Jul 14 11:37:28.810576 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.810157    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8590baa3-db08-49ef-8d26-3f551d6954e4-serviceca\") pod \"node-ca-27fdx\" (UID: \"8590baa3-db08-49ef-8d26-3f551d6954e4\") " pod="openshift-image-registry/node-ca-27fdx"
Jul 14 11:37:28.810576 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.810186    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdhqq\" (UniqueName: \"kubernetes.io/projected/8590baa3-db08-49ef-8d26-3f551d6954e4-kube-api-access-jdhqq\") pod \"node-ca-27fdx\" (UID: \"8590baa3-db08-49ef-8d26-3f551d6954e4\") " pod="openshift-image-registry/node-ca-27fdx"
Jul 14 11:37:28.811298 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.810570    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3bb65b10-6f3c-43ee-8ad9-060bfb4df18e-host-run-netns\") pod \"multus-ll8fh\" (UID: \"3bb65b10-6f3c-43ee-8ad9-060bfb4df18e\") " pod="openshift-multus/multus-ll8fh"
Jul 14 11:37:28.811298 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.810270    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/62c98a11-ece4-4497-99e7-0840a7626a56-ovnkube-config\") pod \"ovnkube-node-cp57b\" (UID: \"62c98a11-ece4-4497-99e7-0840a7626a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cp57b"
Jul 14 11:37:28.811298 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.810583    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4c7434bb-d4dd-407e-a009-33d672d4bfc6-cni-binary-copy\") pod \"multus-additional-cni-plugins-4fhlr\" (UID: \"4c7434bb-d4dd-407e-a009-33d672d4bfc6\") " pod="openshift-multus/multus-additional-cni-plugins-4fhlr"
Jul 14 11:37:28.811298 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.810458    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/62c98a11-ece4-4497-99e7-0840a7626a56-env-overrides\") pod \"ovnkube-node-cp57b\" (UID: \"62c98a11-ece4-4497-99e7-0840a7626a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cp57b"
Jul 14 11:37:28.811298 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.810599    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3bb65b10-6f3c-43ee-8ad9-060bfb4df18e-multus-daemon-config\") pod \"multus-ll8fh\" (UID: \"3bb65b10-6f3c-43ee-8ad9-060bfb4df18e\") " pod="openshift-multus/multus-ll8fh"
Jul 14 11:37:28.811298 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.810467    2551 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 14 11:37:28.811298 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.810629    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3bb65b10-6f3c-43ee-8ad9-060bfb4df18e-host-run-netns\") pod \"multus-ll8fh\" (UID: \"3bb65b10-6f3c-43ee-8ad9-060bfb4df18e\") " pod="openshift-multus/multus-ll8fh"
Jul 14 11:37:28.811298 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.810653    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65mtw\" (UniqueName: \"kubernetes.io/projected/3bb65b10-6f3c-43ee-8ad9-060bfb4df18e-kube-api-access-65mtw\") pod \"multus-ll8fh\" (UID: \"3bb65b10-6f3c-43ee-8ad9-060bfb4df18e\") " pod="openshift-multus/multus-ll8fh"
Jul 14 11:37:28.811298 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.810684    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/62c98a11-ece4-4497-99e7-0840a7626a56-ovnkube-script-lib\") pod \"ovnkube-node-cp57b\" (UID: \"62c98a11-ece4-4497-99e7-0840a7626a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cp57b"
Jul 14 11:37:28.811298 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.810710    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3bb65b10-6f3c-43ee-8ad9-060bfb4df18e-os-release\") pod \"multus-ll8fh\" (UID: \"3bb65b10-6f3c-43ee-8ad9-060bfb4df18e\") " pod="openshift-multus/multus-ll8fh"
Jul 14 11:37:28.811298 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.810739    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3bb65b10-6f3c-43ee-8ad9-060bfb4df18e-host-run-k8s-cni-cncf-io\") pod \"multus-ll8fh\" (UID: \"3bb65b10-6f3c-43ee-8ad9-060bfb4df18e\") " pod="openshift-multus/multus-ll8fh"
Jul 14 11:37:28.811783 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.811385    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3bb65b10-6f3c-43ee-8ad9-060bfb4df18e-multus-daemon-config\") pod \"multus-ll8fh\" (UID: \"3bb65b10-6f3c-43ee-8ad9-060bfb4df18e\") " pod="openshift-multus/multus-ll8fh"
Jul 14 11:37:28.811783 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.811625    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4c7434bb-d4dd-407e-a009-33d672d4bfc6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4fhlr\" (UID: \"4c7434bb-d4dd-407e-a009-33d672d4bfc6\") " pod="openshift-multus/multus-additional-cni-plugins-4fhlr"
Jul 14 11:37:28.811879 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.811780    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3bb65b10-6f3c-43ee-8ad9-060bfb4df18e-os-release\") pod \"multus-ll8fh\" (UID: \"3bb65b10-6f3c-43ee-8ad9-060bfb4df18e\") " pod="openshift-multus/multus-ll8fh"
Jul 14 11:37:28.811879 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.811827    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3bb65b10-6f3c-43ee-8ad9-060bfb4df18e-host-run-k8s-cni-cncf-io\") pod \"multus-ll8fh\" (UID: \"3bb65b10-6f3c-43ee-8ad9-060bfb4df18e\") " pod="openshift-multus/multus-ll8fh"
Jul 14 11:37:28.811972 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.811895    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8590baa3-db08-49ef-8d26-3f551d6954e4-serviceca\") pod \"node-ca-27fdx\" (UID: \"8590baa3-db08-49ef-8d26-3f551d6954e4\") " pod="openshift-image-registry/node-ca-27fdx"
Jul 14 11:37:28.812018 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.811949    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/62c98a11-ece4-4497-99e7-0840a7626a56-ovnkube-script-lib\") pod \"ovnkube-node-cp57b\" (UID: \"62c98a11-ece4-4497-99e7-0840a7626a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cp57b"
Jul 14 11:37:28.815118 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.810767    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4c7434bb-d4dd-407e-a009-33d672d4bfc6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4fhlr\" (UID: \"4c7434bb-d4dd-407e-a009-33d672d4bfc6\") " pod="openshift-multus/multus-additional-cni-plugins-4fhlr"
Jul 14 11:37:28.815118 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.812221    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/62c98a11-ece4-4497-99e7-0840a7626a56-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cp57b\" (UID: \"62c98a11-ece4-4497-99e7-0840a7626a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cp57b"
Jul 14 11:37:28.815118 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.812293    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pqmx\" (UniqueName: \"kubernetes.io/projected/62c98a11-ece4-4497-99e7-0840a7626a56-kube-api-access-5pqmx\") pod \"ovnkube-node-cp57b\" (UID: \"62c98a11-ece4-4497-99e7-0840a7626a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cp57b"
Jul 14 11:37:28.815118 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.812335    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3360b04a-23e1-49f9-84e4-11186ccde5e7-dbus\") pod \"global-pull-secret-syncer-75pwf\" (UID: \"3360b04a-23e1-49f9-84e4-11186ccde5e7\") " pod="kube-system/global-pull-secret-syncer-75pwf"
Jul 14 11:37:28.815118 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.812380    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8590baa3-db08-49ef-8d26-3f551d6954e4-host\") pod \"node-ca-27fdx\" (UID: \"8590baa3-db08-49ef-8d26-3f551d6954e4\") " pod="openshift-image-registry/node-ca-27fdx"
Jul 14 11:37:28.815118 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.812424    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3bb65b10-6f3c-43ee-8ad9-060bfb4df18e-etc-kubernetes\") pod \"multus-ll8fh\" (UID: \"3bb65b10-6f3c-43ee-8ad9-060bfb4df18e\") " pod="openshift-multus/multus-ll8fh"
Jul 14 11:37:28.815118 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.812450    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/62c98a11-ece4-4497-99e7-0840a7626a56-host-run-netns\") pod \"ovnkube-node-cp57b\" (UID: \"62c98a11-ece4-4497-99e7-0840a7626a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cp57b"
Jul 14 11:37:28.815118 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.812476    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1977dd8b-e0ae-4e49-a7d0-ddea4f9a3280-etc-systemd\") pod \"tuned-sjblf\" (UID: \"1977dd8b-e0ae-4e49-a7d0-ddea4f9a3280\") " pod="openshift-cluster-node-tuning-operator/tuned-sjblf"
Jul 14 11:37:28.815118 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.812497    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1977dd8b-e0ae-4e49-a7d0-ddea4f9a3280-etc-tuned\") pod \"tuned-sjblf\" (UID: \"1977dd8b-e0ae-4e49-a7d0-ddea4f9a3280\") " pod="openshift-cluster-node-tuning-operator/tuned-sjblf"
Jul 14 11:37:28.815118 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.812524    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9lbt\" (UniqueName: \"kubernetes.io/projected/846fd65c-288d-4067-9622-6f652e088853-kube-api-access-z9lbt\") pod \"iptables-alerter-rpmgk\" (UID: \"846fd65c-288d-4067-9622-6f652e088853\") " pod="openshift-network-operator/iptables-alerter-rpmgk"
Jul 14 11:37:28.815118 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.812622    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2b7e4c27-d289-460b-b653-a483407a8851-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qm5ql\" (UID: \"2b7e4c27-d289-460b-b653-a483407a8851\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qm5ql"
Jul 14 11:37:28.815118 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.812651    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1977dd8b-e0ae-4e49-a7d0-ddea4f9a3280-etc-sysctl-d\") pod \"tuned-sjblf\" (UID: \"1977dd8b-e0ae-4e49-a7d0-ddea4f9a3280\") " pod="openshift-cluster-node-tuning-operator/tuned-sjblf"
Jul 14 11:37:28.815118 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.812673    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1977dd8b-e0ae-4e49-a7d0-ddea4f9a3280-etc-sysctl-conf\") pod \"tuned-sjblf\" (UID: \"1977dd8b-e0ae-4e49-a7d0-ddea4f9a3280\") " pod="openshift-cluster-node-tuning-operator/tuned-sjblf"
Jul 14 11:37:28.815118 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.812730    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8a3aafb3-0f38-444f-98c4-dbf37fa6effe-konnectivity-ca\") pod \"konnectivity-agent-642k8\" (UID: \"8a3aafb3-0f38-444f-98c4-dbf37fa6effe\") " pod="kube-system/konnectivity-agent-642k8"
Jul 14 11:37:28.815118 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.812757    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2b7e4c27-d289-460b-b653-a483407a8851-socket-dir\") pod \"aws-ebs-csi-driver-node-qm5ql\" (UID: \"2b7e4c27-d289-460b-b653-a483407a8851\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qm5ql"
Jul 14 11:37:28.815118 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.812796    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2b7e4c27-d289-460b-b653-a483407a8851-device-dir\") pod \"aws-ebs-csi-driver-node-qm5ql\" (UID: \"2b7e4c27-d289-460b-b653-a483407a8851\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qm5ql"
Jul 14 11:37:28.815118 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.812836    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3360b04a-23e1-49f9-84e4-11186ccde5e7-original-pull-secret\") pod \"global-pull-secret-syncer-75pwf\" (UID: \"3360b04a-23e1-49f9-84e4-11186ccde5e7\") " pod="kube-system/global-pull-secret-syncer-75pwf"
Jul 14 11:37:28.816072 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.812862    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/846fd65c-288d-4067-9622-6f652e088853-iptables-alerter-script\") pod \"iptables-alerter-rpmgk\" (UID: \"846fd65c-288d-4067-9622-6f652e088853\") " pod="openshift-network-operator/iptables-alerter-rpmgk"
Jul 14 11:37:28.816072 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.812883    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8a3aafb3-0f38-444f-98c4-dbf37fa6effe-agent-certs\") pod \"konnectivity-agent-642k8\" (UID: \"8a3aafb3-0f38-444f-98c4-dbf37fa6effe\") " pod="kube-system/konnectivity-agent-642k8"
Jul 14 11:37:28.816072 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.812937    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e356b532-03eb-4760-93eb-a731dbc0f142-tmp-dir\") pod \"node-resolver-mldq8\" (UID: \"e356b532-03eb-4760-93eb-a731dbc0f142\") " pod="openshift-dns/node-resolver-mldq8"
Jul 14 11:37:28.816072 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.812991    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/62c98a11-ece4-4497-99e7-0840a7626a56-host-slash\") pod \"ovnkube-node-cp57b\" (UID: \"62c98a11-ece4-4497-99e7-0840a7626a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cp57b"
Jul 14 11:37:28.816072 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.813019    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77qmm\" (UniqueName: \"kubernetes.io/projected/2f44b67b-26da-4ab3-9dfa-343ad6334e4f-kube-api-access-77qmm\") pod \"network-check-target-4zvrn\" (UID: \"2f44b67b-26da-4ab3-9dfa-343ad6334e4f\") " pod="openshift-network-diagnostics/network-check-target-4zvrn"
Jul 14 11:37:28.816072 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.813046    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3bb65b10-6f3c-43ee-8ad9-060bfb4df18e-cni-binary-copy\") pod \"multus-ll8fh\" (UID: \"3bb65b10-6f3c-43ee-8ad9-060bfb4df18e\") " pod="openshift-multus/multus-ll8fh"
Jul 14 11:37:28.816072 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.813081    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3bb65b10-6f3c-43ee-8ad9-060bfb4df18e-hostroot\") pod \"multus-ll8fh\" (UID: \"3bb65b10-6f3c-43ee-8ad9-060bfb4df18e\") " pod="openshift-multus/multus-ll8fh"
Jul 14 11:37:28.816072 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.813119    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4c7434bb-d4dd-407e-a009-33d672d4bfc6-cnibin\") pod \"multus-additional-cni-plugins-4fhlr\" (UID: \"4c7434bb-d4dd-407e-a009-33d672d4bfc6\") " pod="openshift-multus/multus-additional-cni-plugins-4fhlr"
Jul 14 11:37:28.816072 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.813159    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4c7434bb-d4dd-407e-a009-33d672d4bfc6-os-release\") pod \"multus-additional-cni-plugins-4fhlr\" (UID: \"4c7434bb-d4dd-407e-a009-33d672d4bfc6\") " pod="openshift-multus/multus-additional-cni-plugins-4fhlr"
Jul 14 11:37:28.816072 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.813306    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/62c98a11-ece4-4497-99e7-0840a7626a56-host-kubelet\") pod \"ovnkube-node-cp57b\" (UID: \"62c98a11-ece4-4497-99e7-0840a7626a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cp57b"
Jul 14 11:37:28.816072 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.813343    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/62c98a11-ece4-4497-99e7-0840a7626a56-etc-openvswitch\") pod \"ovnkube-node-cp57b\" (UID: \"62c98a11-ece4-4497-99e7-0840a7626a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cp57b"
Jul 14 11:37:28.816072 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.813384    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1977dd8b-e0ae-4e49-a7d0-ddea4f9a3280-etc-kubernetes\") pod \"tuned-sjblf\" (UID: \"1977dd8b-e0ae-4e49-a7d0-ddea4f9a3280\") " pod="openshift-cluster-node-tuning-operator/tuned-sjblf"
Jul 14 11:37:28.816072 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.813439    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3bb65b10-6f3c-43ee-8ad9-060bfb4df18e-cnibin\") pod \"multus-ll8fh\" (UID: \"3bb65b10-6f3c-43ee-8ad9-060bfb4df18e\") " pod="openshift-multus/multus-ll8fh"
Jul 14 11:37:28.816072 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.813494    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3bb65b10-6f3c-43ee-8ad9-060bfb4df18e-multus-conf-dir\") pod \"multus-ll8fh\" (UID: \"3bb65b10-6f3c-43ee-8ad9-060bfb4df18e\") " pod="openshift-multus/multus-ll8fh"
Jul 14 11:37:28.816072 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.813536    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/62c98a11-ece4-4497-99e7-0840a7626a56-host-cni-bin\") pod \"ovnkube-node-cp57b\" (UID: \"62c98a11-ece4-4497-99e7-0840a7626a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cp57b"
Jul 14 11:37:28.816072 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.813577    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1977dd8b-e0ae-4e49-a7d0-ddea4f9a3280-var-lib-kubelet\") pod \"tuned-sjblf\" (UID: \"1977dd8b-e0ae-4e49-a7d0-ddea4f9a3280\") " pod="openshift-cluster-node-tuning-operator/tuned-sjblf"
Jul 14 11:37:28.816072 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.813660    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1977dd8b-e0ae-4e49-a7d0-ddea4f9a3280-host\") pod \"tuned-sjblf\" (UID: \"1977dd8b-e0ae-4e49-a7d0-ddea4f9a3280\") " pod="openshift-cluster-node-tuning-operator/tuned-sjblf"
Jul 14 11:37:28.816873 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.813684    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmjhv\" (UniqueName: \"kubernetes.io/projected/2b7e4c27-d289-460b-b653-a483407a8851-kube-api-access-wmjhv\") pod \"aws-ebs-csi-driver-node-qm5ql\" (UID: \"2b7e4c27-d289-460b-b653-a483407a8851\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qm5ql"
Jul 14 11:37:28.816873 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.813712    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3bb65b10-6f3c-43ee-8ad9-060bfb4df18e-host-var-lib-cni-multus\") pod \"multus-ll8fh\" (UID: \"3bb65b10-6f3c-43ee-8ad9-060bfb4df18e\") " pod="openshift-multus/multus-ll8fh"
Jul 14 11:37:28.816873 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.813738    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4c7434bb-d4dd-407e-a009-33d672d4bfc6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4fhlr\" (UID: \"4c7434bb-d4dd-407e-a009-33d672d4bfc6\") " pod="openshift-multus/multus-additional-cni-plugins-4fhlr"
Jul 14 11:37:28.816873 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.813793    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/846fd65c-288d-4067-9622-6f652e088853-host-slash\") pod \"iptables-alerter-rpmgk\" (UID: \"846fd65c-288d-4067-9622-6f652e088853\") " pod="openshift-network-operator/iptables-alerter-rpmgk"
Jul 14 11:37:28.816873 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.813879    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5eafaa1-c985-4204-809a-3e9a52c96e55-metrics-certs\") pod \"network-metrics-daemon-66zbn\" (UID: \"a5eafaa1-c985-4204-809a-3e9a52c96e55\") " pod="openshift-multus/network-metrics-daemon-66zbn"
Jul 14 11:37:28.816873 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.813902    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/62c98a11-ece4-4497-99e7-0840a7626a56-host-run-ovn-kubernetes\") pod \"ovnkube-node-cp57b\" (UID: \"62c98a11-ece4-4497-99e7-0840a7626a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cp57b"
Jul 14 11:37:28.816873 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.813930    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2b7e4c27-d289-460b-b653-a483407a8851-registration-dir\") pod \"aws-ebs-csi-driver-node-qm5ql\" (UID: \"2b7e4c27-d289-460b-b653-a483407a8851\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qm5ql"
Jul 14 11:37:28.816873 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.813985    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2b7e4c27-d289-460b-b653-a483407a8851-etc-selinux\") pod \"aws-ebs-csi-driver-node-qm5ql\" (UID: \"2b7e4c27-d289-460b-b653-a483407a8851\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qm5ql"
Jul 14 11:37:28.816873 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.814121    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3bb65b10-6f3c-43ee-8ad9-060bfb4df18e-host-run-multus-certs\") pod \"multus-ll8fh\" (UID: \"3bb65b10-6f3c-43ee-8ad9-060bfb4df18e\") " pod="openshift-multus/multus-ll8fh"
Jul 14 11:37:28.816873 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.814145    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/62c98a11-ece4-4497-99e7-0840a7626a56-run-openvswitch\") pod \"ovnkube-node-cp57b\" (UID: \"62c98a11-ece4-4497-99e7-0840a7626a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cp57b"
Jul 14 11:37:28.816873 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.814187    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1977dd8b-e0ae-4e49-a7d0-ddea4f9a3280-etc-modprobe-d\") pod \"tuned-sjblf\" (UID: \"1977dd8b-e0ae-4e49-a7d0-ddea4f9a3280\") " pod="openshift-cluster-node-tuning-operator/tuned-sjblf"
Jul 14 11:37:28.816873 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.814213    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1977dd8b-e0ae-4e49-a7d0-ddea4f9a3280-etc-sysconfig\") pod \"tuned-sjblf\" (UID: \"1977dd8b-e0ae-4e49-a7d0-ddea4f9a3280\") " pod="openshift-cluster-node-tuning-operator/tuned-sjblf"
Jul 14 11:37:28.816873 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.814238    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e356b532-03eb-4760-93eb-a731dbc0f142-tmp-dir\") pod \"node-resolver-mldq8\" (UID: \"e356b532-03eb-4760-93eb-a731dbc0f142\") " pod="openshift-dns/node-resolver-mldq8"
Jul 14 11:37:28.816873 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.814238    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1977dd8b-e0ae-4e49-a7d0-ddea4f9a3280-sys\") pod \"tuned-sjblf\" (UID: \"1977dd8b-e0ae-4e49-a7d0-ddea4f9a3280\") " pod="openshift-cluster-node-tuning-operator/tuned-sjblf"
Jul 14 11:37:28.816873 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.814282    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1977dd8b-e0ae-4e49-a7d0-ddea4f9a3280-tmp\") pod \"tuned-sjblf\" (UID: \"1977dd8b-e0ae-4e49-a7d0-ddea4f9a3280\") " pod="openshift-cluster-node-tuning-operator/tuned-sjblf"
Jul 14 11:37:28.816873 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.814328    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3bb65b10-6f3c-43ee-8ad9-060bfb4df18e-host-var-lib-kubelet\") pod \"multus-ll8fh\" (UID: \"3bb65b10-6f3c-43ee-8ad9-060bfb4df18e\") " pod="openshift-multus/multus-ll8fh"
Jul 14 11:37:28.816873 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.814378    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4c7434bb-d4dd-407e-a009-33d672d4bfc6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4fhlr\" (UID: \"4c7434bb-d4dd-407e-a009-33d672d4bfc6\") " pod="openshift-multus/multus-additional-cni-plugins-4fhlr"
Jul 14 11:37:28.817521 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.814454    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mdh8\" (UniqueName: \"kubernetes.io/projected/4c7434bb-d4dd-407e-a009-33d672d4bfc6-kube-api-access-7mdh8\") pod \"multus-additional-cni-plugins-4fhlr\" (UID: \"4c7434bb-d4dd-407e-a009-33d672d4bfc6\") " pod="openshift-multus/multus-additional-cni-plugins-4fhlr"
Jul 14 11:37:28.817521 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.814707    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3360b04a-23e1-49f9-84e4-11186ccde5e7-dbus\") pod \"global-pull-secret-syncer-75pwf\" (UID: \"3360b04a-23e1-49f9-84e4-11186ccde5e7\") " pod="kube-system/global-pull-secret-syncer-75pwf"
Jul 14 11:37:28.817521 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.814723    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/62c98a11-ece4-4497-99e7-0840a7626a56-host-slash\") pod \"ovnkube-node-cp57b\" (UID: \"62c98a11-ece4-4497-99e7-0840a7626a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cp57b"
Jul 14 11:37:28.817521 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.814744    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3bb65b10-6f3c-43ee-8ad9-060bfb4df18e-host-var-lib-cni-multus\") pod \"multus-ll8fh\" (UID: \"3bb65b10-6f3c-43ee-8ad9-060bfb4df18e\") " pod="openshift-multus/multus-ll8fh"
Jul 14 11:37:28.817521 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.814785    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8590baa3-db08-49ef-8d26-3f551d6954e4-host\") pod \"node-ca-27fdx\" (UID: \"8590baa3-db08-49ef-8d26-3f551d6954e4\") " pod="openshift-image-registry/node-ca-27fdx"
Jul 14 11:37:28.817521 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.814856    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/62c98a11-ece4-4497-99e7-0840a7626a56-host-run-netns\") pod \"ovnkube-node-cp57b\" (UID: \"62c98a11-ece4-4497-99e7-0840a7626a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cp57b"
Jul 14 11:37:28.817521 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.814879    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/846fd65c-288d-4067-9622-6f652e088853-host-slash\") pod \"iptables-alerter-rpmgk\" (UID: \"846fd65c-288d-4067-9622-6f652e088853\") " pod="openshift-network-operator/iptables-alerter-rpmgk"
Jul 14 11:37:28.817521 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.814901    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/62c98a11-ece4-4497-99e7-0840a7626a56-run-openvswitch\") pod \"ovnkube-node-cp57b\" (UID: \"62c98a11-ece4-4497-99e7-0840a7626a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cp57b"
Jul 14 11:37:28.817521 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.814969    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1977dd8b-e0ae-4e49-a7d0-ddea4f9a3280-etc-systemd\") pod \"tuned-sjblf\" (UID: \"1977dd8b-e0ae-4e49-a7d0-ddea4f9a3280\") " pod="openshift-cluster-node-tuning-operator/tuned-sjblf"
Jul 14 11:37:28.817521 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.815060    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/62c98a11-ece4-4497-99e7-0840a7626a56-host-run-ovn-kubernetes\") pod \"ovnkube-node-cp57b\" (UID: \"62c98a11-ece4-4497-99e7-0840a7626a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cp57b"
Jul 14 11:37:28.817521 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.814338    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/62c98a11-ece4-4497-99e7-0840a7626a56-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cp57b\" (UID: \"62c98a11-ece4-4497-99e7-0840a7626a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cp57b"
Jul 14 11:37:28.817521 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.815341    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4c7434bb-d4dd-407e-a009-33d672d4bfc6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4fhlr\" (UID: \"4c7434bb-d4dd-407e-a009-33d672d4bfc6\") " pod="openshift-multus/multus-additional-cni-plugins-4fhlr"
Jul 14 11:37:28.817521 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.815405    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4c7434bb-d4dd-407e-a009-33d672d4bfc6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4fhlr\" (UID: \"4c7434bb-d4dd-407e-a009-33d672d4bfc6\") " pod="openshift-multus/multus-additional-cni-plugins-4fhlr"
Jul 14 11:37:28.817521 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:28.815474    2551 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 11:37:28.817521 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:28.815588    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5eafaa1-c985-4204-809a-3e9a52c96e55-metrics-certs podName:a5eafaa1-c985-4204-809a-3e9a52c96e55 nodeName:}" failed. No retries permitted until 2026-07-14 11:37:29.315538924 +0000 UTC m=+1.991535406 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a5eafaa1-c985-4204-809a-3e9a52c96e55-metrics-certs") pod "network-metrics-daemon-66zbn" (UID: "a5eafaa1-c985-4204-809a-3e9a52c96e55") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 11:37:28.817521 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.815684    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3bb65b10-6f3c-43ee-8ad9-060bfb4df18e-cni-binary-copy\") pod \"multus-ll8fh\" (UID: \"3bb65b10-6f3c-43ee-8ad9-060bfb4df18e\") " pod="openshift-multus/multus-ll8fh"
Jul 14 11:37:28.817521 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.815689    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3bb65b10-6f3c-43ee-8ad9-060bfb4df18e-hostroot\") pod \"multus-ll8fh\" (UID: \"3bb65b10-6f3c-43ee-8ad9-060bfb4df18e\") " pod="openshift-multus/multus-ll8fh"
Jul 14 11:37:28.818331 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.815705    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3bb65b10-6f3c-43ee-8ad9-060bfb4df18e-etc-kubernetes\") pod \"multus-ll8fh\" (UID: \"3bb65b10-6f3c-43ee-8ad9-060bfb4df18e\") " pod="openshift-multus/multus-ll8fh"
Jul 14 11:37:28.818331 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.815684    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2b7e4c27-d289-460b-b653-a483407a8851-registration-dir\") pod \"aws-ebs-csi-driver-node-qm5ql\" (UID: \"2b7e4c27-d289-460b-b653-a483407a8851\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qm5ql"
Jul 14 11:37:28.818331 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.815824    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2b7e4c27-d289-460b-b653-a483407a8851-socket-dir\") pod \"aws-ebs-csi-driver-node-qm5ql\" (UID: \"2b7e4c27-d289-460b-b653-a483407a8851\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qm5ql"
Jul 14 11:37:28.818331 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.815888    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/62c98a11-ece4-4497-99e7-0840a7626a56-host-kubelet\") pod \"ovnkube-node-cp57b\" (UID: \"62c98a11-ece4-4497-99e7-0840a7626a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cp57b"
Jul 14 11:37:28.818331 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.815898    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3bb65b10-6f3c-43ee-8ad9-060bfb4df18e-multus-conf-dir\") pod \"multus-ll8fh\" (UID: \"3bb65b10-6f3c-43ee-8ad9-060bfb4df18e\") " pod="openshift-multus/multus-ll8fh"
Jul 14 11:37:28.818331 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.815940    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1977dd8b-e0ae-4e49-a7d0-ddea4f9a3280-host\") pod \"tuned-sjblf\" (UID: \"1977dd8b-e0ae-4e49-a7d0-ddea4f9a3280\") " pod="openshift-cluster-node-tuning-operator/tuned-sjblf"
Jul 14 11:37:28.818331 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.815948    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/62c98a11-ece4-4497-99e7-0840a7626a56-host-cni-bin\") pod \"ovnkube-node-cp57b\" (UID: \"62c98a11-ece4-4497-99e7-0840a7626a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cp57b"
Jul 14 11:37:28.818331 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.816007    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1977dd8b-e0ae-4e49-a7d0-ddea4f9a3280-etc-kubernetes\") pod \"tuned-sjblf\" (UID: \"1977dd8b-e0ae-4e49-a7d0-ddea4f9a3280\") " pod="openshift-cluster-node-tuning-operator/tuned-sjblf"
Jul 14 11:37:28.818331 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.816045    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2b7e4c27-d289-460b-b653-a483407a8851-device-dir\") pod \"aws-ebs-csi-driver-node-qm5ql\" (UID: \"2b7e4c27-d289-460b-b653-a483407a8851\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qm5ql"
Jul 14 11:37:28.818331 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.816055    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/62c98a11-ece4-4497-99e7-0840a7626a56-etc-openvswitch\") pod \"ovnkube-node-cp57b\" (UID: \"62c98a11-ece4-4497-99e7-0840a7626a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cp57b"
Jul 14 11:37:28.818331 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.816087    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1977dd8b-e0ae-4e49-a7d0-ddea4f9a3280-etc-modprobe-d\") pod \"tuned-sjblf\" (UID: \"1977dd8b-e0ae-4e49-a7d0-ddea4f9a3280\") " pod="openshift-cluster-node-tuning-operator/tuned-sjblf"
Jul 14 11:37:28.818331 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.816091    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/62c98a11-ece4-4497-99e7-0840a7626a56-run-ovn\") pod \"ovnkube-node-cp57b\" (UID: \"62c98a11-ece4-4497-99e7-0840a7626a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cp57b"
Jul 14 11:37:28.818331 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.816124    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2b7e4c27-d289-460b-b653-a483407a8851-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qm5ql\" (UID: \"2b7e4c27-d289-460b-b653-a483407a8851\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qm5ql"
Jul 14 11:37:28.818331 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.816128    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3bb65b10-6f3c-43ee-8ad9-060bfb4df18e-cnibin\") pod \"multus-ll8fh\" (UID: \"3bb65b10-6f3c-43ee-8ad9-060bfb4df18e\") " pod="openshift-multus/multus-ll8fh"
Jul 14 11:37:28.818331 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.816125    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/62c98a11-ece4-4497-99e7-0840a7626a56-log-socket\") pod \"ovnkube-node-cp57b\" (UID: \"62c98a11-ece4-4497-99e7-0840a7626a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cp57b"
Jul 14 11:37:28.818331 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.816172    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gnnl\" (UniqueName: \"kubernetes.io/projected/1977dd8b-e0ae-4e49-a7d0-ddea4f9a3280-kube-api-access-5gnnl\") pod \"tuned-sjblf\" (UID: \"1977dd8b-e0ae-4e49-a7d0-ddea4f9a3280\") " pod="openshift-cluster-node-tuning-operator/tuned-sjblf"
Jul 14 11:37:28.818331 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.816196    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/62c98a11-ece4-4497-99e7-0840a7626a56-run-ovn\") pod \"ovnkube-node-cp57b\" (UID: \"62c98a11-ece4-4497-99e7-0840a7626a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cp57b"
Jul 14 11:37:28.818331 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.816145    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1977dd8b-e0ae-4e49-a7d0-ddea4f9a3280-var-lib-kubelet\") pod \"tuned-sjblf\" (UID: \"1977dd8b-e0ae-4e49-a7d0-ddea4f9a3280\") " pod="openshift-cluster-node-tuning-operator/tuned-sjblf"
Jul 14 11:37:28.819255 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.816231    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1977dd8b-e0ae-4e49-a7d0-ddea4f9a3280-etc-sysctl-d\") pod \"tuned-sjblf\" (UID: \"1977dd8b-e0ae-4e49-a7d0-ddea4f9a3280\") " pod="openshift-cluster-node-tuning-operator/tuned-sjblf"
Jul 14 11:37:28.819255 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:28.816318    2551 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 14 11:37:28.819255 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.816320    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3bb65b10-6f3c-43ee-8ad9-060bfb4df18e-host-run-multus-certs\") pod \"multus-ll8fh\" (UID: \"3bb65b10-6f3c-43ee-8ad9-060bfb4df18e\") " pod="openshift-multus/multus-ll8fh"
Jul 14 11:37:28.819255 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.816332    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3bb65b10-6f3c-43ee-8ad9-060bfb4df18e-host-var-lib-kubelet\") pod \"multus-ll8fh\" (UID: \"3bb65b10-6f3c-43ee-8ad9-060bfb4df18e\") " pod="openshift-multus/multus-ll8fh"
Jul 14 11:37:28.819255 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:28.816382    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3360b04a-23e1-49f9-84e4-11186ccde5e7-original-pull-secret podName:3360b04a-23e1-49f9-84e4-11186ccde5e7 nodeName:}" failed. No retries permitted until 2026-07-14 11:37:29.316365329 +0000 UTC m=+1.992361805 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3360b04a-23e1-49f9-84e4-11186ccde5e7-original-pull-secret") pod "global-pull-secret-syncer-75pwf" (UID: "3360b04a-23e1-49f9-84e4-11186ccde5e7") : object "kube-system"/"original-pull-secret" not registered
Jul 14 11:37:28.819255 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.816408    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1977dd8b-e0ae-4e49-a7d0-ddea4f9a3280-etc-sysconfig\") pod \"tuned-sjblf\" (UID: \"1977dd8b-e0ae-4e49-a7d0-ddea4f9a3280\") " pod="openshift-cluster-node-tuning-operator/tuned-sjblf"
Jul 14 11:37:28.819255 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.816511    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4c7434bb-d4dd-407e-a009-33d672d4bfc6-os-release\") pod \"multus-additional-cni-plugins-4fhlr\" (UID: \"4c7434bb-d4dd-407e-a009-33d672d4bfc6\") " pod="openshift-multus/multus-additional-cni-plugins-4fhlr"
Jul 14 11:37:28.819255 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.816533    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1977dd8b-e0ae-4e49-a7d0-ddea4f9a3280-sys\") pod \"tuned-sjblf\" (UID: \"1977dd8b-e0ae-4e49-a7d0-ddea4f9a3280\") " pod="openshift-cluster-node-tuning-operator/tuned-sjblf"
Jul 14 11:37:28.819255 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.816565    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/62c98a11-ece4-4497-99e7-0840a7626a56-log-socket\") pod \"ovnkube-node-cp57b\" (UID: \"62c98a11-ece4-4497-99e7-0840a7626a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cp57b"
Jul 14 11:37:28.819255 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.816692    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1977dd8b-e0ae-4e49-a7d0-ddea4f9a3280-etc-sysctl-conf\") pod \"tuned-sjblf\" (UID: \"1977dd8b-e0ae-4e49-a7d0-ddea4f9a3280\") " pod="openshift-cluster-node-tuning-operator/tuned-sjblf"
Jul 14 11:37:28.819255 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.816721    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8a3aafb3-0f38-444f-98c4-dbf37fa6effe-konnectivity-ca\") pod \"konnectivity-agent-642k8\" (UID: \"8a3aafb3-0f38-444f-98c4-dbf37fa6effe\") " pod="kube-system/konnectivity-agent-642k8"
Jul 14 11:37:28.819255 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.816865    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4c7434bb-d4dd-407e-a009-33d672d4bfc6-cnibin\") pod \"multus-additional-cni-plugins-4fhlr\" (UID: \"4c7434bb-d4dd-407e-a009-33d672d4bfc6\") " pod="openshift-multus/multus-additional-cni-plugins-4fhlr"
Jul 14 11:37:28.819255 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.816926    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2b7e4c27-d289-460b-b653-a483407a8851-etc-selinux\") pod \"aws-ebs-csi-driver-node-qm5ql\" (UID: \"2b7e4c27-d289-460b-b653-a483407a8851\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qm5ql"
Jul 14 11:37:28.819255 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.817062    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/846fd65c-288d-4067-9622-6f652e088853-iptables-alerter-script\") pod \"iptables-alerter-rpmgk\" (UID: \"846fd65c-288d-4067-9622-6f652e088853\") " pod="openshift-network-operator/iptables-alerter-rpmgk"
Jul 14 11:37:28.819255 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.817116    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/62c98a11-ece4-4497-99e7-0840a7626a56-ovn-node-metrics-cert\") pod \"ovnkube-node-cp57b\" (UID: \"62c98a11-ece4-4497-99e7-0840a7626a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cp57b"
Jul 14 11:37:28.819255 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.817818    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1977dd8b-e0ae-4e49-a7d0-ddea4f9a3280-etc-tuned\") pod \"tuned-sjblf\" (UID: \"1977dd8b-e0ae-4e49-a7d0-ddea4f9a3280\") " pod="openshift-cluster-node-tuning-operator/tuned-sjblf"
Jul 14 11:37:28.819255 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.818512    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1977dd8b-e0ae-4e49-a7d0-ddea4f9a3280-tmp\") pod \"tuned-sjblf\" (UID: \"1977dd8b-e0ae-4e49-a7d0-ddea4f9a3280\") " pod="openshift-cluster-node-tuning-operator/tuned-sjblf"
Jul 14 11:37:28.820187 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.819181    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8a3aafb3-0f38-444f-98c4-dbf37fa6effe-agent-certs\") pod \"konnectivity-agent-642k8\" (UID: \"8a3aafb3-0f38-444f-98c4-dbf37fa6effe\") " pod="kube-system/konnectivity-agent-642k8"
Jul 14 11:37:28.820415 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.820378    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65mtw\" (UniqueName: \"kubernetes.io/projected/3bb65b10-6f3c-43ee-8ad9-060bfb4df18e-kube-api-access-65mtw\") pod \"multus-ll8fh\" (UID: \"3bb65b10-6f3c-43ee-8ad9-060bfb4df18e\") " pod="openshift-multus/multus-ll8fh"
Jul 14 11:37:28.820663 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.820640    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdhqq\" (UniqueName: \"kubernetes.io/projected/8590baa3-db08-49ef-8d26-3f551d6954e4-kube-api-access-jdhqq\") pod \"node-ca-27fdx\" (UID: \"8590baa3-db08-49ef-8d26-3f551d6954e4\") " pod="openshift-image-registry/node-ca-27fdx"
Jul 14 11:37:28.820758 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.820742    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k86gd\" (UniqueName: \"kubernetes.io/projected/a5eafaa1-c985-4204-809a-3e9a52c96e55-kube-api-access-k86gd\") pod \"network-metrics-daemon-66zbn\" (UID: \"a5eafaa1-c985-4204-809a-3e9a52c96e55\") " pod="openshift-multus/network-metrics-daemon-66zbn"
Jul 14 11:37:28.821013 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.820993    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffnvs\" (UniqueName: \"kubernetes.io/projected/e356b532-03eb-4760-93eb-a731dbc0f142-kube-api-access-ffnvs\") pod \"node-resolver-mldq8\" (UID: \"e356b532-03eb-4760-93eb-a731dbc0f142\") " pod="openshift-dns/node-resolver-mldq8"
Jul 14 11:37:28.825109 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:28.825089    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 14 11:37:28.825109 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:28.825108    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 14 11:37:28.825263 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:28.825120    2551 projected.go:196] Error preparing data for projected volume kube-api-access-77qmm for pod openshift-network-diagnostics/network-check-target-4zvrn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 11:37:28.825263 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:28.825172    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2f44b67b-26da-4ab3-9dfa-343ad6334e4f-kube-api-access-77qmm podName:2f44b67b-26da-4ab3-9dfa-343ad6334e4f nodeName:}" failed. No retries permitted until 2026-07-14 11:37:29.325157943 +0000 UTC m=+2.001154404 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-77qmm" (UniqueName: "kubernetes.io/projected/2f44b67b-26da-4ab3-9dfa-343ad6334e4f-kube-api-access-77qmm") pod "network-check-target-4zvrn" (UID: "2f44b67b-26da-4ab3-9dfa-343ad6334e4f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 11:37:28.827086 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.827056    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmjhv\" (UniqueName: \"kubernetes.io/projected/2b7e4c27-d289-460b-b653-a483407a8851-kube-api-access-wmjhv\") pod \"aws-ebs-csi-driver-node-qm5ql\" (UID: \"2b7e4c27-d289-460b-b653-a483407a8851\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qm5ql"
Jul 14 11:37:28.827229 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.827161    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pqmx\" (UniqueName: \"kubernetes.io/projected/62c98a11-ece4-4497-99e7-0840a7626a56-kube-api-access-5pqmx\") pod \"ovnkube-node-cp57b\" (UID: \"62c98a11-ece4-4497-99e7-0840a7626a56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cp57b"
Jul 14 11:37:28.828606 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.828587    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mdh8\" (UniqueName: \"kubernetes.io/projected/4c7434bb-d4dd-407e-a009-33d672d4bfc6-kube-api-access-7mdh8\") pod \"multus-additional-cni-plugins-4fhlr\" (UID: \"4c7434bb-d4dd-407e-a009-33d672d4bfc6\") " pod="openshift-multus/multus-additional-cni-plugins-4fhlr"
Jul 14 11:37:28.829145 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.829127    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9lbt\" (UniqueName: \"kubernetes.io/projected/846fd65c-288d-4067-9622-6f652e088853-kube-api-access-z9lbt\") pod \"iptables-alerter-rpmgk\" (UID: \"846fd65c-288d-4067-9622-6f652e088853\") " pod="openshift-network-operator/iptables-alerter-rpmgk"
Jul 14 11:37:28.829215 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.829139    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gnnl\" (UniqueName: \"kubernetes.io/projected/1977dd8b-e0ae-4e49-a7d0-ddea4f9a3280-kube-api-access-5gnnl\") pod \"tuned-sjblf\" (UID: \"1977dd8b-e0ae-4e49-a7d0-ddea4f9a3280\") " pod="openshift-cluster-node-tuning-operator/tuned-sjblf"
Jul 14 11:37:28.928221 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.928128    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mldq8"
Jul 14 11:37:28.938463 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:28.938436    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode356b532_03eb_4760_93eb_a731dbc0f142.slice/crio-b545e77e2647ebe4d44ed0d942b8bfc62e98a550a868ffae1aa668ad180db13f WatchSource:0}: Error finding container b545e77e2647ebe4d44ed0d942b8bfc62e98a550a868ffae1aa668ad180db13f: Status 404 returned error can't find the container with id b545e77e2647ebe4d44ed0d942b8bfc62e98a550a868ffae1aa668ad180db13f
Jul 14 11:37:28.945430 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.945409    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-27fdx"
Jul 14 11:37:28.953681 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:28.953652    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8590baa3_db08_49ef_8d26_3f551d6954e4.slice/crio-58c80421864a71e0aa12914b9b6af469eeb00eb49df79e3f53fa8b07d5100bba WatchSource:0}: Error finding container 58c80421864a71e0aa12914b9b6af469eeb00eb49df79e3f53fa8b07d5100bba: Status 404 returned error can't find the container with id 58c80421864a71e0aa12914b9b6af469eeb00eb49df79e3f53fa8b07d5100bba
Jul 14 11:37:28.962931 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.962907    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ll8fh"
Jul 14 11:37:28.967973 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.967953    2551 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 14 11:37:28.972027 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:28.972001    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bb65b10_6f3c_43ee_8ad9_060bfb4df18e.slice/crio-53787de62d4e0087cad46667570b291260e94ac1d09b07d721858efbe704b635 WatchSource:0}: Error finding container 53787de62d4e0087cad46667570b291260e94ac1d09b07d721858efbe704b635: Status 404 returned error can't find the container with id 53787de62d4e0087cad46667570b291260e94ac1d09b07d721858efbe704b635
Jul 14 11:37:28.986960 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:28.986924    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rpmgk"
Jul 14 11:37:28.995240 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:28.995210    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod846fd65c_288d_4067_9622_6f652e088853.slice/crio-87171e345a465e193baf47ee2587e64e7c70a7c67fb00d107da0c03bcac42e67 WatchSource:0}: Error finding container 87171e345a465e193baf47ee2587e64e7c70a7c67fb00d107da0c03bcac42e67: Status 404 returned error can't find the container with id 87171e345a465e193baf47ee2587e64e7c70a7c67fb00d107da0c03bcac42e67
Jul 14 11:37:29.002266 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:29.002246    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-642k8"
Jul 14 11:37:29.011467 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:29.011443    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a3aafb3_0f38_444f_98c4_dbf37fa6effe.slice/crio-0f71c52470b282946d312663b40f1227a6772ac5af8689e41164e13327e874f1 WatchSource:0}: Error finding container 0f71c52470b282946d312663b40f1227a6772ac5af8689e41164e13327e874f1: Status 404 returned error can't find the container with id 0f71c52470b282946d312663b40f1227a6772ac5af8689e41164e13327e874f1
Jul 14 11:37:29.023234 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:29.023211    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qm5ql"
Jul 14 11:37:29.028204 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:29.028185    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4fhlr"
Jul 14 11:37:29.032697 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:29.032669    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b7e4c27_d289_460b_b653_a483407a8851.slice/crio-37bb272fd5b7b4ad4a322aeece1765e36a99b55fcf8ef6d7c58d2a87a0e445e6 WatchSource:0}: Error finding container 37bb272fd5b7b4ad4a322aeece1765e36a99b55fcf8ef6d7c58d2a87a0e445e6: Status 404 returned error can't find the container with id 37bb272fd5b7b4ad4a322aeece1765e36a99b55fcf8ef6d7c58d2a87a0e445e6
Jul 14 11:37:29.033485 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:29.033468    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-cp57b"
Jul 14 11:37:29.037746 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:29.037714    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c7434bb_d4dd_407e_a009_33d672d4bfc6.slice/crio-616e2837d306cfbc18cd0e3588b14fc3767afd069e0446654fd55d22ca670b22 WatchSource:0}: Error finding container 616e2837d306cfbc18cd0e3588b14fc3767afd069e0446654fd55d22ca670b22: Status 404 returned error can't find the container with id 616e2837d306cfbc18cd0e3588b14fc3767afd069e0446654fd55d22ca670b22
Jul 14 11:37:29.038355 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:29.038337    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-sjblf"
Jul 14 11:37:29.045814 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:29.045787    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62c98a11_ece4_4497_99e7_0840a7626a56.slice/crio-698e9f9cbee439bbf1d5e53545759b91cbf1dc04d5c18fea9272585a06bdb460 WatchSource:0}: Error finding container 698e9f9cbee439bbf1d5e53545759b91cbf1dc04d5c18fea9272585a06bdb460: Status 404 returned error can't find the container with id 698e9f9cbee439bbf1d5e53545759b91cbf1dc04d5c18fea9272585a06bdb460
Jul 14 11:37:29.048708 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:37:29.048689    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1977dd8b_e0ae_4e49_a7d0_ddea4f9a3280.slice/crio-f47fc003ac3e32de6e268de91497ae357dca3551555f98fb3616bb5315cfab45 WatchSource:0}: Error finding container f47fc003ac3e32de6e268de91497ae357dca3551555f98fb3616bb5315cfab45: Status 404 returned error can't find the container with id f47fc003ac3e32de6e268de91497ae357dca3551555f98fb3616bb5315cfab45
Jul 14 11:37:30.764887 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:30.764852    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5eafaa1-c985-4204-809a-3e9a52c96e55-metrics-certs\") pod \"network-metrics-daemon-66zbn\" (UID: \"a5eafaa1-c985-4204-809a-3e9a52c96e55\") " pod="openshift-multus/network-metrics-daemon-66zbn"
Jul 14 11:37:30.765067 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:30.765013    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3360b04a-23e1-49f9-84e4-11186ccde5e7-original-pull-secret\") pod \"global-pull-secret-syncer-75pwf\" (UID: \"3360b04a-23e1-49f9-84e4-11186ccde5e7\") " pod="kube-system/global-pull-secret-syncer-75pwf"
Jul 14 11:37:30.765135 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:30.765030    2551 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 11:37:30.765183 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:30.765151    2551 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 14 11:37:30.765231 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:30.765212    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3360b04a-23e1-49f9-84e4-11186ccde5e7-original-pull-secret podName:3360b04a-23e1-49f9-84e4-11186ccde5e7 nodeName:}" failed. No retries permitted until 2026-07-14 11:37:31.765191951 +0000 UTC m=+2.997001241 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3360b04a-23e1-49f9-84e4-11186ccde5e7-original-pull-secret") pod "global-pull-secret-syncer-75pwf" (UID: "3360b04a-23e1-49f9-84e4-11186ccde5e7") : object "kube-system"/"original-pull-secret" not registered
Jul 14 11:37:30.765292 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:30.765240    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5eafaa1-c985-4204-809a-3e9a52c96e55-metrics-certs podName:a5eafaa1-c985-4204-809a-3e9a52c96e55 nodeName:}" failed. No retries permitted until 2026-07-14 11:37:31.765219451 +0000 UTC m=+2.997028742 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a5eafaa1-c985-4204-809a-3e9a52c96e55-metrics-certs") pod "network-metrics-daemon-66zbn" (UID: "a5eafaa1-c985-4204-809a-3e9a52c96e55") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 11:37:30.866248 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:30.866095    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77qmm\" (UniqueName: \"kubernetes.io/projected/2f44b67b-26da-4ab3-9dfa-343ad6334e4f-kube-api-access-77qmm\") pod \"network-check-target-4zvrn\" (UID: \"2f44b67b-26da-4ab3-9dfa-343ad6334e4f\") " pod="openshift-network-diagnostics/network-check-target-4zvrn"
Jul 14 11:37:30.866417 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:30.866273    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 14 11:37:30.866417 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:30.866292    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 14 11:37:30.866417 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:30.866304    2551 projected.go:196] Error preparing data for projected volume kube-api-access-77qmm for pod openshift-network-diagnostics/network-check-target-4zvrn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 11:37:30.866417 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:30.866366    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2f44b67b-26da-4ab3-9dfa-343ad6334e4f-kube-api-access-77qmm podName:2f44b67b-26da-4ab3-9dfa-343ad6334e4f nodeName:}" failed. No retries permitted until 2026-07-14 11:37:31.866347548 +0000 UTC m=+3.098156843 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-77qmm" (UniqueName: "kubernetes.io/projected/2f44b67b-26da-4ab3-9dfa-343ad6334e4f-kube-api-access-77qmm") pod "network-check-target-4zvrn" (UID: "2f44b67b-26da-4ab3-9dfa-343ad6334e4f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 11:37:30.929420 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:30.929361    2551 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 14 11:37:31.102768 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:31.102671    2551 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-13 11:32:30 +0000 UTC" deadline="2028-03-10 15:30:13.412411694 +0000 UTC"
Jul 14 11:37:31.102768 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:31.102715    2551 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14523h52m42.309701242s"
Jul 14 11:37:31.187802 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:31.187766    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4zvrn"
Jul 14 11:37:31.187983 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:31.187903    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4zvrn" podUID="2f44b67b-26da-4ab3-9dfa-343ad6334e4f"
Jul 14 11:37:31.201303 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:31.201265    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-sjblf" event={"ID":"1977dd8b-e0ae-4e49-a7d0-ddea4f9a3280","Type":"ContainerStarted","Data":"f47fc003ac3e32de6e268de91497ae357dca3551555f98fb3616bb5315cfab45"}
Jul 14 11:37:31.209553 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:31.209510    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4fhlr" event={"ID":"4c7434bb-d4dd-407e-a009-33d672d4bfc6","Type":"ContainerStarted","Data":"616e2837d306cfbc18cd0e3588b14fc3767afd069e0446654fd55d22ca670b22"}
Jul 14 11:37:31.219413 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:31.219328    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-642k8" event={"ID":"8a3aafb3-0f38-444f-98c4-dbf37fa6effe","Type":"ContainerStarted","Data":"0f71c52470b282946d312663b40f1227a6772ac5af8689e41164e13327e874f1"}
Jul 14 11:37:31.239081 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:31.239037    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rpmgk" event={"ID":"846fd65c-288d-4067-9622-6f652e088853","Type":"ContainerStarted","Data":"87171e345a465e193baf47ee2587e64e7c70a7c67fb00d107da0c03bcac42e67"}
Jul 14 11:37:31.250570 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:31.250423    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ll8fh" event={"ID":"3bb65b10-6f3c-43ee-8ad9-060bfb4df18e","Type":"ContainerStarted","Data":"53787de62d4e0087cad46667570b291260e94ac1d09b07d721858efbe704b635"}
Jul 14 11:37:31.262812 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:31.262767    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-27fdx" event={"ID":"8590baa3-db08-49ef-8d26-3f551d6954e4","Type":"ContainerStarted","Data":"58c80421864a71e0aa12914b9b6af469eeb00eb49df79e3f53fa8b07d5100bba"}
Jul 14 11:37:31.275530 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:31.275490    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mldq8" event={"ID":"e356b532-03eb-4760-93eb-a731dbc0f142","Type":"ContainerStarted","Data":"b545e77e2647ebe4d44ed0d942b8bfc62e98a550a868ffae1aa668ad180db13f"}
Jul 14 11:37:31.293622 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:31.293568    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cp57b" event={"ID":"62c98a11-ece4-4497-99e7-0840a7626a56","Type":"ContainerStarted","Data":"698e9f9cbee439bbf1d5e53545759b91cbf1dc04d5c18fea9272585a06bdb460"}
Jul 14 11:37:31.307264 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:31.307219    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qm5ql" event={"ID":"2b7e4c27-d289-460b-b653-a483407a8851","Type":"ContainerStarted","Data":"37bb272fd5b7b4ad4a322aeece1765e36a99b55fcf8ef6d7c58d2a87a0e445e6"}
Jul 14 11:37:31.775002 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:31.774955    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3360b04a-23e1-49f9-84e4-11186ccde5e7-original-pull-secret\") pod \"global-pull-secret-syncer-75pwf\" (UID: \"3360b04a-23e1-49f9-84e4-11186ccde5e7\") " pod="kube-system/global-pull-secret-syncer-75pwf"
Jul 14 11:37:31.775192 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:31.775040    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5eafaa1-c985-4204-809a-3e9a52c96e55-metrics-certs\") pod \"network-metrics-daemon-66zbn\" (UID: \"a5eafaa1-c985-4204-809a-3e9a52c96e55\") " pod="openshift-multus/network-metrics-daemon-66zbn"
Jul 14 11:37:31.775256 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:31.775189    2551 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 11:37:31.775256 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:31.775252    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5eafaa1-c985-4204-809a-3e9a52c96e55-metrics-certs podName:a5eafaa1-c985-4204-809a-3e9a52c96e55 nodeName:}" failed. No retries permitted until 2026-07-14 11:37:33.775232767 +0000 UTC m=+5.007042044 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a5eafaa1-c985-4204-809a-3e9a52c96e55-metrics-certs") pod "network-metrics-daemon-66zbn" (UID: "a5eafaa1-c985-4204-809a-3e9a52c96e55") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 11:37:31.775708 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:31.775688    2551 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 14 11:37:31.775808 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:31.775739    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3360b04a-23e1-49f9-84e4-11186ccde5e7-original-pull-secret podName:3360b04a-23e1-49f9-84e4-11186ccde5e7 nodeName:}" failed. No retries permitted until 2026-07-14 11:37:33.775723898 +0000 UTC m=+5.007533177 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3360b04a-23e1-49f9-84e4-11186ccde5e7-original-pull-secret") pod "global-pull-secret-syncer-75pwf" (UID: "3360b04a-23e1-49f9-84e4-11186ccde5e7") : object "kube-system"/"original-pull-secret" not registered
Jul 14 11:37:31.875896 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:31.875858    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77qmm\" (UniqueName: \"kubernetes.io/projected/2f44b67b-26da-4ab3-9dfa-343ad6334e4f-kube-api-access-77qmm\") pod \"network-check-target-4zvrn\" (UID: \"2f44b67b-26da-4ab3-9dfa-343ad6334e4f\") " pod="openshift-network-diagnostics/network-check-target-4zvrn"
Jul 14 11:37:31.876130 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:31.876082    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 14 11:37:31.876130 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:31.876108    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 14 11:37:31.876130 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:31.876121    2551 projected.go:196] Error preparing data for projected volume kube-api-access-77qmm for pod openshift-network-diagnostics/network-check-target-4zvrn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 11:37:31.876328 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:31.876183    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2f44b67b-26da-4ab3-9dfa-343ad6334e4f-kube-api-access-77qmm podName:2f44b67b-26da-4ab3-9dfa-343ad6334e4f nodeName:}" failed. No retries permitted until 2026-07-14 11:37:33.876164031 +0000 UTC m=+5.107973320 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-77qmm" (UniqueName: "kubernetes.io/projected/2f44b67b-26da-4ab3-9dfa-343ad6334e4f-kube-api-access-77qmm") pod "network-check-target-4zvrn" (UID: "2f44b67b-26da-4ab3-9dfa-343ad6334e4f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 11:37:32.104032 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:32.103895    2551 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-13 11:32:30 +0000 UTC" deadline="2028-04-29 22:07:11.30661496 +0000 UTC"
Jul 14 11:37:32.104032 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:32.103953    2551 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15730h29m39.202665987s"
Jul 14 11:37:32.186216 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:32.185427    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-75pwf"
Jul 14 11:37:32.186216 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:32.185625    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-75pwf" podUID="3360b04a-23e1-49f9-84e4-11186ccde5e7"
Jul 14 11:37:32.186216 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:32.186054    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-66zbn"
Jul 14 11:37:32.186216 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:32.186163    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-66zbn" podUID="a5eafaa1-c985-4204-809a-3e9a52c96e55"
Jul 14 11:37:32.229621 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:32.229394    2551 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 14 11:37:32.814061 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:32.814031    2551 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 14 11:37:33.187472 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:33.187394    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4zvrn"
Jul 14 11:37:33.187995 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:33.187522    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4zvrn" podUID="2f44b67b-26da-4ab3-9dfa-343ad6334e4f"
Jul 14 11:37:33.793469 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:33.793431    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5eafaa1-c985-4204-809a-3e9a52c96e55-metrics-certs\") pod \"network-metrics-daemon-66zbn\" (UID: \"a5eafaa1-c985-4204-809a-3e9a52c96e55\") " pod="openshift-multus/network-metrics-daemon-66zbn"
Jul 14 11:37:33.793778 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:33.793524    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3360b04a-23e1-49f9-84e4-11186ccde5e7-original-pull-secret\") pod \"global-pull-secret-syncer-75pwf\" (UID: \"3360b04a-23e1-49f9-84e4-11186ccde5e7\") " pod="kube-system/global-pull-secret-syncer-75pwf"
Jul 14 11:37:33.793778 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:33.793677    2551 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 14 11:37:33.793778 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:33.793682    2551 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 11:37:33.793778 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:33.793746    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3360b04a-23e1-49f9-84e4-11186ccde5e7-original-pull-secret podName:3360b04a-23e1-49f9-84e4-11186ccde5e7 nodeName:}" failed. No retries permitted until 2026-07-14 11:37:37.793725068 +0000 UTC m=+9.025534345 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3360b04a-23e1-49f9-84e4-11186ccde5e7-original-pull-secret") pod "global-pull-secret-syncer-75pwf" (UID: "3360b04a-23e1-49f9-84e4-11186ccde5e7") : object "kube-system"/"original-pull-secret" not registered
Jul 14 11:37:33.793778 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:33.793767    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5eafaa1-c985-4204-809a-3e9a52c96e55-metrics-certs podName:a5eafaa1-c985-4204-809a-3e9a52c96e55 nodeName:}" failed. No retries permitted until 2026-07-14 11:37:37.793756696 +0000 UTC m=+9.025565974 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a5eafaa1-c985-4204-809a-3e9a52c96e55-metrics-certs") pod "network-metrics-daemon-66zbn" (UID: "a5eafaa1-c985-4204-809a-3e9a52c96e55") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 11:37:33.895554 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:33.894901    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77qmm\" (UniqueName: \"kubernetes.io/projected/2f44b67b-26da-4ab3-9dfa-343ad6334e4f-kube-api-access-77qmm\") pod \"network-check-target-4zvrn\" (UID: \"2f44b67b-26da-4ab3-9dfa-343ad6334e4f\") " pod="openshift-network-diagnostics/network-check-target-4zvrn"
Jul 14 11:37:33.895554 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:33.895113    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 14 11:37:33.895554 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:33.895132    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 14 11:37:33.895554 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:33.895145    2551 projected.go:196] Error preparing data for projected volume kube-api-access-77qmm for pod openshift-network-diagnostics/network-check-target-4zvrn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 11:37:33.895554 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:33.895204    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2f44b67b-26da-4ab3-9dfa-343ad6334e4f-kube-api-access-77qmm podName:2f44b67b-26da-4ab3-9dfa-343ad6334e4f nodeName:}" failed. No retries permitted until 2026-07-14 11:37:37.895185176 +0000 UTC m=+9.126994455 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-77qmm" (UniqueName: "kubernetes.io/projected/2f44b67b-26da-4ab3-9dfa-343ad6334e4f-kube-api-access-77qmm") pod "network-check-target-4zvrn" (UID: "2f44b67b-26da-4ab3-9dfa-343ad6334e4f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 11:37:34.185610 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:34.185436    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-75pwf"
Jul 14 11:37:34.185610 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:34.185456    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-66zbn"
Jul 14 11:37:34.185831 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:34.185603    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-75pwf" podUID="3360b04a-23e1-49f9-84e4-11186ccde5e7"
Jul 14 11:37:34.185831 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:34.185701    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-66zbn" podUID="a5eafaa1-c985-4204-809a-3e9a52c96e55"
Jul 14 11:37:35.186122 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:35.186081    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4zvrn"
Jul 14 11:37:35.186563 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:35.186232    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4zvrn" podUID="2f44b67b-26da-4ab3-9dfa-343ad6334e4f"
Jul 14 11:37:36.185771 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:36.185264    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-66zbn"
Jul 14 11:37:36.185771 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:36.185311    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-75pwf"
Jul 14 11:37:36.185771 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:36.185415    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-66zbn" podUID="a5eafaa1-c985-4204-809a-3e9a52c96e55"
Jul 14 11:37:36.185771 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:36.185698    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-75pwf" podUID="3360b04a-23e1-49f9-84e4-11186ccde5e7"
Jul 14 11:37:37.185398 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:37.184821    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4zvrn"
Jul 14 11:37:37.185398 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:37.184958    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4zvrn" podUID="2f44b67b-26da-4ab3-9dfa-343ad6334e4f"
Jul 14 11:37:37.827693 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:37.827652    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5eafaa1-c985-4204-809a-3e9a52c96e55-metrics-certs\") pod \"network-metrics-daemon-66zbn\" (UID: \"a5eafaa1-c985-4204-809a-3e9a52c96e55\") " pod="openshift-multus/network-metrics-daemon-66zbn"
Jul 14 11:37:37.827883 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:37.827737    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3360b04a-23e1-49f9-84e4-11186ccde5e7-original-pull-secret\") pod \"global-pull-secret-syncer-75pwf\" (UID: \"3360b04a-23e1-49f9-84e4-11186ccde5e7\") " pod="kube-system/global-pull-secret-syncer-75pwf"
Jul 14 11:37:37.827883 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:37.827817    2551 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 11:37:37.827883 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:37.827840    2551 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 14 11:37:37.828063 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:37.827905    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5eafaa1-c985-4204-809a-3e9a52c96e55-metrics-certs podName:a5eafaa1-c985-4204-809a-3e9a52c96e55 nodeName:}" failed. No retries permitted until 2026-07-14 11:37:45.827882214 +0000 UTC m=+17.059691505 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a5eafaa1-c985-4204-809a-3e9a52c96e55-metrics-certs") pod "network-metrics-daemon-66zbn" (UID: "a5eafaa1-c985-4204-809a-3e9a52c96e55") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 11:37:37.828063 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:37.827928    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3360b04a-23e1-49f9-84e4-11186ccde5e7-original-pull-secret podName:3360b04a-23e1-49f9-84e4-11186ccde5e7 nodeName:}" failed. No retries permitted until 2026-07-14 11:37:45.827916995 +0000 UTC m=+17.059726271 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3360b04a-23e1-49f9-84e4-11186ccde5e7-original-pull-secret") pod "global-pull-secret-syncer-75pwf" (UID: "3360b04a-23e1-49f9-84e4-11186ccde5e7") : object "kube-system"/"original-pull-secret" not registered
Jul 14 11:37:37.928852 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:37.928811    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77qmm\" (UniqueName: \"kubernetes.io/projected/2f44b67b-26da-4ab3-9dfa-343ad6334e4f-kube-api-access-77qmm\") pod \"network-check-target-4zvrn\" (UID: \"2f44b67b-26da-4ab3-9dfa-343ad6334e4f\") " pod="openshift-network-diagnostics/network-check-target-4zvrn"
Jul 14 11:37:37.929052 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:37.928957    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 14 11:37:37.929052 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:37.928981    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 14 11:37:37.929052 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:37.928996    2551 projected.go:196] Error preparing data for projected volume kube-api-access-77qmm for pod openshift-network-diagnostics/network-check-target-4zvrn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 11:37:37.929204 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:37.929111    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2f44b67b-26da-4ab3-9dfa-343ad6334e4f-kube-api-access-77qmm podName:2f44b67b-26da-4ab3-9dfa-343ad6334e4f nodeName:}" failed. No retries permitted until 2026-07-14 11:37:45.929077579 +0000 UTC m=+17.160886868 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-77qmm" (UniqueName: "kubernetes.io/projected/2f44b67b-26da-4ab3-9dfa-343ad6334e4f-kube-api-access-77qmm") pod "network-check-target-4zvrn" (UID: "2f44b67b-26da-4ab3-9dfa-343ad6334e4f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 11:37:38.185615 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:38.185064    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-75pwf"
Jul 14 11:37:38.185615 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:38.185200    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-75pwf" podUID="3360b04a-23e1-49f9-84e4-11186ccde5e7"
Jul 14 11:37:38.185615 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:38.185354    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-66zbn"
Jul 14 11:37:38.185615 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:38.185500    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-66zbn" podUID="a5eafaa1-c985-4204-809a-3e9a52c96e55"
Jul 14 11:37:39.187147 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:39.186647    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4zvrn"
Jul 14 11:37:39.187147 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:39.186770    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4zvrn" podUID="2f44b67b-26da-4ab3-9dfa-343ad6334e4f"
Jul 14 11:37:40.186240 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:40.186135    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-75pwf"
Jul 14 11:37:40.186240 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:40.186189    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-66zbn"
Jul 14 11:37:40.186630 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:40.186568    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-75pwf" podUID="3360b04a-23e1-49f9-84e4-11186ccde5e7"
Jul 14 11:37:40.186777 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:40.186705    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-66zbn" podUID="a5eafaa1-c985-4204-809a-3e9a52c96e55"
Jul 14 11:37:41.184846 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:41.184796    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4zvrn"
Jul 14 11:37:41.185206 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:41.184926    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4zvrn" podUID="2f44b67b-26da-4ab3-9dfa-343ad6334e4f"
Jul 14 11:37:42.185051 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:42.184996    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-66zbn"
Jul 14 11:37:42.185481 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:42.185020    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-75pwf"
Jul 14 11:37:42.185481 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:42.185155    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-66zbn" podUID="a5eafaa1-c985-4204-809a-3e9a52c96e55"
Jul 14 11:37:42.185481 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:42.185229    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-75pwf" podUID="3360b04a-23e1-49f9-84e4-11186ccde5e7"
Jul 14 11:37:43.184807 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:43.184756    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4zvrn"
Jul 14 11:37:43.185011 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:43.184906    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4zvrn" podUID="2f44b67b-26da-4ab3-9dfa-343ad6334e4f"
Jul 14 11:37:44.185288 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:44.185068    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-66zbn"
Jul 14 11:37:44.185740 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:44.185087    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-75pwf"
Jul 14 11:37:44.185740 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:44.185409    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-66zbn" podUID="a5eafaa1-c985-4204-809a-3e9a52c96e55"
Jul 14 11:37:44.185848 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:44.185469    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-75pwf" podUID="3360b04a-23e1-49f9-84e4-11186ccde5e7"
Jul 14 11:37:45.184912 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:45.184869    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4zvrn"
Jul 14 11:37:45.185109 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:45.184995    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4zvrn" podUID="2f44b67b-26da-4ab3-9dfa-343ad6334e4f"
Jul 14 11:37:45.892804 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:45.892763    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5eafaa1-c985-4204-809a-3e9a52c96e55-metrics-certs\") pod \"network-metrics-daemon-66zbn\" (UID: \"a5eafaa1-c985-4204-809a-3e9a52c96e55\") " pod="openshift-multus/network-metrics-daemon-66zbn"
Jul 14 11:37:45.893299 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:45.892852    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3360b04a-23e1-49f9-84e4-11186ccde5e7-original-pull-secret\") pod \"global-pull-secret-syncer-75pwf\" (UID: \"3360b04a-23e1-49f9-84e4-11186ccde5e7\") " pod="kube-system/global-pull-secret-syncer-75pwf"
Jul 14 11:37:45.893299 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:45.892935    2551 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 11:37:45.893299 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:45.892953    2551 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 14 11:37:45.893299 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:45.893021    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3360b04a-23e1-49f9-84e4-11186ccde5e7-original-pull-secret podName:3360b04a-23e1-49f9-84e4-11186ccde5e7 nodeName:}" failed. No retries permitted until 2026-07-14 11:38:01.892998768 +0000 UTC m=+33.124808109 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3360b04a-23e1-49f9-84e4-11186ccde5e7-original-pull-secret") pod "global-pull-secret-syncer-75pwf" (UID: "3360b04a-23e1-49f9-84e4-11186ccde5e7") : object "kube-system"/"original-pull-secret" not registered
Jul 14 11:37:45.893299 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:45.893043    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5eafaa1-c985-4204-809a-3e9a52c96e55-metrics-certs podName:a5eafaa1-c985-4204-809a-3e9a52c96e55 nodeName:}" failed. No retries permitted until 2026-07-14 11:38:01.893032468 +0000 UTC m=+33.124841745 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a5eafaa1-c985-4204-809a-3e9a52c96e55-metrics-certs") pod "network-metrics-daemon-66zbn" (UID: "a5eafaa1-c985-4204-809a-3e9a52c96e55") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 11:37:45.993432 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:45.993392    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77qmm\" (UniqueName: \"kubernetes.io/projected/2f44b67b-26da-4ab3-9dfa-343ad6334e4f-kube-api-access-77qmm\") pod \"network-check-target-4zvrn\" (UID: \"2f44b67b-26da-4ab3-9dfa-343ad6334e4f\") " pod="openshift-network-diagnostics/network-check-target-4zvrn"
Jul 14 11:37:45.993596 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:45.993523    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 14 11:37:45.993596 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:45.993547    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 14 11:37:45.993596 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:45.993558    2551 projected.go:196] Error preparing data for projected volume kube-api-access-77qmm for pod openshift-network-diagnostics/network-check-target-4zvrn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 11:37:45.993715 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:45.993628    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2f44b67b-26da-4ab3-9dfa-343ad6334e4f-kube-api-access-77qmm podName:2f44b67b-26da-4ab3-9dfa-343ad6334e4f nodeName:}" failed. No retries permitted until 2026-07-14 11:38:01.99361356 +0000 UTC m=+33.225422834 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-77qmm" (UniqueName: "kubernetes.io/projected/2f44b67b-26da-4ab3-9dfa-343ad6334e4f-kube-api-access-77qmm") pod "network-check-target-4zvrn" (UID: "2f44b67b-26da-4ab3-9dfa-343ad6334e4f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 11:37:46.185658 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:46.185562    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-75pwf"
Jul 14 11:37:46.185829 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:46.185561    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-66zbn"
Jul 14 11:37:46.185829 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:46.185696    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-75pwf" podUID="3360b04a-23e1-49f9-84e4-11186ccde5e7"
Jul 14 11:37:46.185933 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:46.185827    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-66zbn" podUID="a5eafaa1-c985-4204-809a-3e9a52c96e55"
Jul 14 11:37:47.185269 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:47.185220    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4zvrn"
Jul 14 11:37:47.185851 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:47.185354    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4zvrn" podUID="2f44b67b-26da-4ab3-9dfa-343ad6334e4f"
Jul 14 11:37:48.185766 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:48.185727    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-66zbn"
Jul 14 11:37:48.185766 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:48.185765    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-75pwf"
Jul 14 11:37:48.186297 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:48.185883    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-66zbn" podUID="a5eafaa1-c985-4204-809a-3e9a52c96e55"
Jul 14 11:37:48.186297 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:48.186005    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-75pwf" podUID="3360b04a-23e1-49f9-84e4-11186ccde5e7"
Jul 14 11:37:49.185783 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:49.185747    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4zvrn"
Jul 14 11:37:49.186253 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:49.185856    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4zvrn" podUID="2f44b67b-26da-4ab3-9dfa-343ad6334e4f"
Jul 14 11:37:50.185408 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:50.185371    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-66zbn"
Jul 14 11:37:50.185593 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:50.185412    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-75pwf"
Jul 14 11:37:50.185593 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:50.185487    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-66zbn" podUID="a5eafaa1-c985-4204-809a-3e9a52c96e55"
Jul 14 11:37:50.185677 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:50.185614    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-75pwf" podUID="3360b04a-23e1-49f9-84e4-11186ccde5e7"
Jul 14 11:37:51.185463 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:51.185325    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4zvrn"
Jul 14 11:37:51.186034 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:51.185542    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4zvrn" podUID="2f44b67b-26da-4ab3-9dfa-343ad6334e4f"
Jul 14 11:37:51.357610 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:51.357389    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-173.ec2.internal" event={"ID":"a552e96d3a56282f33a24bc1bc8a195b","Type":"ContainerStarted","Data":"c1931a432c443ea71619ea967e2c00f056f39219da8c493419073b026a996212"}
Jul 14 11:37:51.359676 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:51.359657    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cp57b_62c98a11-ece4-4497-99e7-0840a7626a56/ovn-acl-logging/0.log"
Jul 14 11:37:51.360015 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:51.359994    2551 generic.go:356] "Generic (PLEG): container finished" podID="62c98a11-ece4-4497-99e7-0840a7626a56" containerID="d3cf6a60aff291c55e1b33a784bfa689abf450c4926eda8270b8672d04da79a0" exitCode=1
Jul 14 11:37:51.360092 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:51.360060    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cp57b" event={"ID":"62c98a11-ece4-4497-99e7-0840a7626a56","Type":"ContainerStarted","Data":"bc9ee011850191adf61fc04051539a3b47eb2067b02e7533fe54aea60db884ce"}
Jul 14 11:37:51.360151 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:51.360131    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cp57b" event={"ID":"62c98a11-ece4-4497-99e7-0840a7626a56","Type":"ContainerStarted","Data":"c84f73f0f122dba395fb125fb202bec5bc2bc7e2ad8acad4f9c6260dad0ebc41"}
Jul 14 11:37:51.360202 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:51.360177    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cp57b" event={"ID":"62c98a11-ece4-4497-99e7-0840a7626a56","Type":"ContainerStarted","Data":"de91eb73a176e6b0053c136a61369294f018326a1ea038f558f7988df9802f04"}
Jul 14 11:37:51.360252 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:51.360206    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cp57b" event={"ID":"62c98a11-ece4-4497-99e7-0840a7626a56","Type":"ContainerDied","Data":"d3cf6a60aff291c55e1b33a784bfa689abf450c4926eda8270b8672d04da79a0"}
Jul 14 11:37:51.360252 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:51.360241    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cp57b" event={"ID":"62c98a11-ece4-4497-99e7-0840a7626a56","Type":"ContainerStarted","Data":"8821628e96f4aec1b8e78df1dc1a9df811baabce08b16d6030a9d7e3cdd11947"}
Jul 14 11:37:51.361415 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:51.361393    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qm5ql" event={"ID":"2b7e4c27-d289-460b-b653-a483407a8851","Type":"ContainerStarted","Data":"6dff420fa4a5becf3f054daee62e34f5f808740ebf6886d56fe87b2ee5a12bd5"}
Jul 14 11:37:51.362697 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:51.362678    2551 generic.go:356] "Generic (PLEG): container finished" podID="6283d946e4456c5cef54b2da21896045" containerID="933ce00cbb415f5c8d46551bbfc028c6d50edaba930f3d42feb53e8baa6fd269" exitCode=0
Jul 14 11:37:51.362787 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:51.362737    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-173.ec2.internal" event={"ID":"6283d946e4456c5cef54b2da21896045","Type":"ContainerDied","Data":"933ce00cbb415f5c8d46551bbfc028c6d50edaba930f3d42feb53e8baa6fd269"}
Jul 14 11:37:51.364157 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:51.364139    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-sjblf" event={"ID":"1977dd8b-e0ae-4e49-a7d0-ddea4f9a3280","Type":"ContainerStarted","Data":"10ef01df0c1891453b0c455c9a15ae17af9afa75707ead99b60e8dce8dd47e8f"}
Jul 14 11:37:51.367919 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:51.367895    2551 generic.go:356] "Generic (PLEG): container finished" podID="4c7434bb-d4dd-407e-a009-33d672d4bfc6" containerID="55d0c9e5c7fb702e52fa422f6a57921809458753063e66002bded02015b7aef3" exitCode=0
Jul 14 11:37:51.368014 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:51.367968    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4fhlr" event={"ID":"4c7434bb-d4dd-407e-a009-33d672d4bfc6","Type":"ContainerDied","Data":"55d0c9e5c7fb702e52fa422f6a57921809458753063e66002bded02015b7aef3"}
Jul 14 11:37:51.369343 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:51.369319    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-642k8" event={"ID":"8a3aafb3-0f38-444f-98c4-dbf37fa6effe","Type":"ContainerStarted","Data":"a300853ac8dbefcace145a43ddd4072679e5ee86738662bc6d853595092ec000"}
Jul 14 11:37:51.370779 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:51.370752    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ll8fh" event={"ID":"3bb65b10-6f3c-43ee-8ad9-060bfb4df18e","Type":"ContainerStarted","Data":"de2533584907835192142fffd0c295b328788239464050fc9e064d72e653bb2c"}
Jul 14 11:37:51.372146 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:51.372121    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-27fdx" event={"ID":"8590baa3-db08-49ef-8d26-3f551d6954e4","Type":"ContainerStarted","Data":"ccb70af7091b1bb59d392eb01a20185984ae5fbf81c9f88586e38f1a2dcaeff7"}
Jul 14 11:37:51.373337 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:51.373318    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mldq8" event={"ID":"e356b532-03eb-4760-93eb-a731dbc0f142","Type":"ContainerStarted","Data":"deab77bba3635127feb4914bd36336072fa682de4b4ce417b26de96e000cf020"}
Jul 14 11:37:51.417993 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:51.417946    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-sjblf" podStartSLOduration=2.271388928 podStartE2EDuration="22.41792203s" totalImagesPullingTime="20.146533102s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 11:37:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 11:37:51.402407556 +0000 UTC m=+22.634216852" watchObservedRunningTime="2026-07-14 11:37:51.41792203 +0000 UTC m=+22.649731304"
Jul 14 11:37:51.418683 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:51.418655    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-173.ec2.internal" podStartSLOduration=22.41864707 podStartE2EDuration="22.41864707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 11:37:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 11:37:51.418036588 +0000 UTC m=+22.649845884" watchObservedRunningTime="2026-07-14 11:37:51.41864707 +0000 UTC m=+22.650456367"
Jul 14 11:37:51.443541 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:51.443476    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-mldq8" podStartSLOduration=2.118502528 podStartE2EDuration="22.443456929s" totalImagesPullingTime="20.324954401s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 11:37:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 11:37:51.443104993 +0000 UTC m=+22.674914290" watchObservedRunningTime="2026-07-14 11:37:51.443456929 +0000 UTC m=+22.675266233"
Jul 14 11:37:51.465821 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:51.465771    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-ll8fh" podStartSLOduration=2.077868469 podStartE2EDuration="22.46575595s" totalImagesPullingTime="20.387887481s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 11:37:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 11:37:51.465245589 +0000 UTC m=+22.697054884" watchObservedRunningTime="2026-07-14 11:37:51.46575595 +0000 UTC m=+22.697565246"
Jul 14 11:37:51.481629 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:51.481566    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-642k8" podStartSLOduration=2.228616664 podStartE2EDuration="22.481549889s" totalImagesPullingTime="20.252933225s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 11:37:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 11:37:51.481480889 +0000 UTC m=+22.713290184" watchObservedRunningTime="2026-07-14 11:37:51.481549889 +0000 UTC m=+22.713359186"
Jul 14 11:37:51.525058 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:51.525003    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-27fdx" podStartSLOduration=2.215471637 podStartE2EDuration="22.524982073s" totalImagesPullingTime="20.309510436s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 11:37:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 11:37:51.524245411 +0000 UTC m=+22.756054706" watchObservedRunningTime="2026-07-14 11:37:51.524982073 +0000 UTC m=+22.756791406"
Jul 14 11:37:52.185036 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:52.184997    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-75pwf"
Jul 14 11:37:52.185223 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:52.185003    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-66zbn"
Jul 14 11:37:52.185223 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:52.185116    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-75pwf" podUID="3360b04a-23e1-49f9-84e4-11186ccde5e7"
Jul 14 11:37:52.185223 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:52.185213    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-66zbn" podUID="a5eafaa1-c985-4204-809a-3e9a52c96e55"
Jul 14 11:37:52.378749 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:52.378723    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cp57b_62c98a11-ece4-4497-99e7-0840a7626a56/ovn-acl-logging/0.log"
Jul 14 11:37:52.379156 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:52.379078    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cp57b" event={"ID":"62c98a11-ece4-4497-99e7-0840a7626a56","Type":"ContainerStarted","Data":"81224f4dd4ac082f4602d3a1336ac02f1d347962209793499b075b017cb94627"}
Jul 14 11:37:52.380731 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:52.380703    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-173.ec2.internal" event={"ID":"6283d946e4456c5cef54b2da21896045","Type":"ContainerStarted","Data":"441d64ad8825baa604d0766ea2be8249b498fe85a3acae0d1663c625f08ac691"}
Jul 14 11:37:52.382095 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:52.382047    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rpmgk" event={"ID":"846fd65c-288d-4067-9622-6f652e088853","Type":"ContainerStarted","Data":"8203e19868b184c79510c812899e646ba5bcbc2ace5de53a54970b2fa916f19b"}
Jul 14 11:37:52.399993 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:52.399888    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-173.ec2.internal" podStartSLOduration=23.399868382 podStartE2EDuration="23.399868382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 11:37:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 11:37:52.399564962 +0000 UTC m=+23.631374259" watchObservedRunningTime="2026-07-14 11:37:52.399868382 +0000 UTC m=+23.631677679"
Jul 14 11:37:52.416620 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:52.416569    2551 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 14 11:37:52.417562 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:52.417512    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-rpmgk" podStartSLOduration=3.2176102 podStartE2EDuration="23.417493349s" totalImagesPullingTime="20.199883149s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 11:37:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 11:37:52.417379787 +0000 UTC m=+23.649189078" watchObservedRunningTime="2026-07-14 11:37:52.417493349 +0000 UTC m=+23.649302646"
Jul 14 11:37:53.126639 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:53.126503    2551 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-14T11:37:52.416610574Z","UUID":"260ed1d6-2832-4826-91cb-1dbffad18b63","Handler":null,"Name":"","Endpoint":""}
Jul 14 11:37:53.128303 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:53.128278    2551 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 14 11:37:53.128444 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:53.128323    2551 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 14 11:37:53.185849 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:53.185809    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4zvrn"
Jul 14 11:37:53.186009 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:53.185951    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4zvrn" podUID="2f44b67b-26da-4ab3-9dfa-343ad6334e4f"
Jul 14 11:37:53.387260 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:53.387179    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qm5ql" event={"ID":"2b7e4c27-d289-460b-b653-a483407a8851","Type":"ContainerStarted","Data":"f059787b18c7c861e6beba06f140f2ec48318ed2c9efcc3d25bb68e663598009"}
Jul 14 11:37:53.430197 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:53.430153    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-642k8"
Jul 14 11:37:53.430788 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:53.430765    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-642k8"
Jul 14 11:37:54.185203 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:54.185167    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-66zbn"
Jul 14 11:37:54.185539 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:54.185211    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-75pwf"
Jul 14 11:37:54.185539 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:54.185310    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-66zbn" podUID="a5eafaa1-c985-4204-809a-3e9a52c96e55"
Jul 14 11:37:54.185539 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:54.185432    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-75pwf" podUID="3360b04a-23e1-49f9-84e4-11186ccde5e7"
Jul 14 11:37:54.391945 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:54.391917    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cp57b_62c98a11-ece4-4497-99e7-0840a7626a56/ovn-acl-logging/0.log"
Jul 14 11:37:54.392447 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:54.392318    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cp57b" event={"ID":"62c98a11-ece4-4497-99e7-0840a7626a56","Type":"ContainerStarted","Data":"e8bde0878ca262c7e67c9f37a5ae34a548fe70acc7c54df8ffcf9f38bb7cd58d"}
Jul 14 11:37:54.394263 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:54.394232    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qm5ql" event={"ID":"2b7e4c27-d289-460b-b653-a483407a8851","Type":"ContainerStarted","Data":"f90f56361c46a9d03e04a2b11b5ce95b67039fb5d4a72cfba71e06f052fb8ff5"}
Jul 14 11:37:54.416192 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:54.416140    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qm5ql" podStartSLOduration=2.925414058 podStartE2EDuration="25.416123473s" totalImagesPullingTime="22.490709415s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 11:37:29 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 11:37:54.415805601 +0000 UTC m=+25.647614898" watchObservedRunningTime="2026-07-14 11:37:54.416123473 +0000 UTC m=+25.647932769"
Jul 14 11:37:55.185202 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:55.184943    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4zvrn"
Jul 14 11:37:55.185424 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:55.185332    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4zvrn" podUID="2f44b67b-26da-4ab3-9dfa-343ad6334e4f"
Jul 14 11:37:55.860658 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:55.860624    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-642k8"
Jul 14 11:37:55.861249 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:55.860764    2551 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 14 11:37:55.861309 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:55.861269    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-642k8"
Jul 14 11:37:56.185116 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:56.184937    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-75pwf"
Jul 14 11:37:56.185259 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:56.185019    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-66zbn"
Jul 14 11:37:56.185259 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:56.185199    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-75pwf" podUID="3360b04a-23e1-49f9-84e4-11186ccde5e7"
Jul 14 11:37:56.185324 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:56.185295    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-66zbn" podUID="a5eafaa1-c985-4204-809a-3e9a52c96e55"
Jul 14 11:37:56.400460 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:56.400429    2551 generic.go:356] "Generic (PLEG): container finished" podID="4c7434bb-d4dd-407e-a009-33d672d4bfc6" containerID="102dbb184f2d91e31cf918574e6dc7ddb6fe74b6bd27cceda0694312d788247e" exitCode=0
Jul 14 11:37:56.400647 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:56.400517    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4fhlr" event={"ID":"4c7434bb-d4dd-407e-a009-33d672d4bfc6","Type":"ContainerDied","Data":"102dbb184f2d91e31cf918574e6dc7ddb6fe74b6bd27cceda0694312d788247e"}
Jul 14 11:37:56.407263 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:56.407235    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cp57b_62c98a11-ece4-4497-99e7-0840a7626a56/ovn-acl-logging/0.log"
Jul 14 11:37:56.407765 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:56.407733    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cp57b" event={"ID":"62c98a11-ece4-4497-99e7-0840a7626a56","Type":"ContainerStarted","Data":"6f104671b0b8e6f8366974c45ceadc4237866824a90e548ec150e974dbed93e0"}
Jul 14 11:37:56.408184 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:56.408165    2551 scope.go:117] "RemoveContainer" containerID="d3cf6a60aff291c55e1b33a784bfa689abf450c4926eda8270b8672d04da79a0"
Jul 14 11:37:57.185852 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:57.185819    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4zvrn"
Jul 14 11:37:57.186635 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:57.185956    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4zvrn" podUID="2f44b67b-26da-4ab3-9dfa-343ad6334e4f"
Jul 14 11:37:57.414186 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:57.414109    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cp57b_62c98a11-ece4-4497-99e7-0840a7626a56/ovn-acl-logging/0.log"
Jul 14 11:37:57.414523 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:57.414499    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cp57b" event={"ID":"62c98a11-ece4-4497-99e7-0840a7626a56","Type":"ContainerStarted","Data":"cf04b8ab7c623fac611d4ec53b96bc868b54b42b9a27d0272b88a804ef9e49c0"}
Jul 14 11:37:57.414639 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:57.414610    2551 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 14 11:37:57.414768 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:57.414745    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cp57b"
Jul 14 11:37:57.414889 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:57.414877    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cp57b"
Jul 14 11:37:57.432063 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:57.432034    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cp57b"
Jul 14 11:37:57.432203 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:57.432140    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cp57b"
Jul 14 11:37:57.451307 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:57.451257    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-cp57b" podStartSLOduration=8.187581787 podStartE2EDuration="28.451239647s" totalImagesPullingTime="20.26365786s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 11:37:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 11:37:57.450820831 +0000 UTC m=+28.682630145" watchObservedRunningTime="2026-07-14 11:37:57.451239647 +0000 UTC m=+28.683048968"
Jul 14 11:37:57.974300 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:57.974257    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-66zbn"]
Jul 14 11:37:57.974572 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:57.974412    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-66zbn"
Jul 14 11:37:57.974572 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:57.974530    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-66zbn" podUID="a5eafaa1-c985-4204-809a-3e9a52c96e55"
Jul 14 11:37:57.976719 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:57.976694    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-75pwf"]
Jul 14 11:37:57.976854 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:57.976798    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-75pwf"
Jul 14 11:37:57.976915 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:57.976899    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-75pwf" podUID="3360b04a-23e1-49f9-84e4-11186ccde5e7"
Jul 14 11:37:57.990855 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:57.990824    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-4zvrn"]
Jul 14 11:37:57.991044 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:57.990936    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4zvrn"
Jul 14 11:37:57.991044 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:57.991022    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4zvrn" podUID="2f44b67b-26da-4ab3-9dfa-343ad6334e4f"
Jul 14 11:37:58.418931 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:58.418838    2551 generic.go:356] "Generic (PLEG): container finished" podID="4c7434bb-d4dd-407e-a009-33d672d4bfc6" containerID="838d270305835c40615a968c24ef3402f4064d1005b9b6a1ebfa034e07d16b46" exitCode=0
Jul 14 11:37:58.419471 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:58.418919    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4fhlr" event={"ID":"4c7434bb-d4dd-407e-a009-33d672d4bfc6","Type":"ContainerDied","Data":"838d270305835c40615a968c24ef3402f4064d1005b9b6a1ebfa034e07d16b46"}
Jul 14 11:37:58.419471 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:58.418998    2551 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 14 11:37:58.628724 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:58.628691    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cp57b"
Jul 14 11:37:59.185757 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:37:59.185721    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-75pwf"
Jul 14 11:37:59.185922 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:37:59.185816    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-75pwf" podUID="3360b04a-23e1-49f9-84e4-11186ccde5e7"
Jul 14 11:38:00.184996 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:00.184966    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4zvrn"
Jul 14 11:38:00.185342 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:38:00.185072    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4zvrn" podUID="2f44b67b-26da-4ab3-9dfa-343ad6334e4f"
Jul 14 11:38:00.185342 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:00.185122    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-66zbn"
Jul 14 11:38:00.185342 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:38:00.185229    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-66zbn" podUID="a5eafaa1-c985-4204-809a-3e9a52c96e55"
Jul 14 11:38:00.426208 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:00.426111    2551 generic.go:356] "Generic (PLEG): container finished" podID="4c7434bb-d4dd-407e-a009-33d672d4bfc6" containerID="ad29a3c395764fe4a0a5d1545e2b3ee60ff72006ffd735449d6fb0cdb54f3c93" exitCode=0
Jul 14 11:38:00.426208 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:00.426193    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4fhlr" event={"ID":"4c7434bb-d4dd-407e-a009-33d672d4bfc6","Type":"ContainerDied","Data":"ad29a3c395764fe4a0a5d1545e2b3ee60ff72006ffd735449d6fb0cdb54f3c93"}
Jul 14 11:38:01.185837 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:01.185800    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-75pwf"
Jul 14 11:38:01.186312 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:38:01.185911    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-75pwf" podUID="3360b04a-23e1-49f9-84e4-11186ccde5e7"
Jul 14 11:38:01.904654 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:01.904547    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3360b04a-23e1-49f9-84e4-11186ccde5e7-original-pull-secret\") pod \"global-pull-secret-syncer-75pwf\" (UID: \"3360b04a-23e1-49f9-84e4-11186ccde5e7\") " pod="kube-system/global-pull-secret-syncer-75pwf"
Jul 14 11:38:01.904823 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:01.904655    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5eafaa1-c985-4204-809a-3e9a52c96e55-metrics-certs\") pod \"network-metrics-daemon-66zbn\" (UID: \"a5eafaa1-c985-4204-809a-3e9a52c96e55\") " pod="openshift-multus/network-metrics-daemon-66zbn"
Jul 14 11:38:01.904823 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:38:01.904726    2551 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 14 11:38:01.904823 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:38:01.904774    2551 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 11:38:01.904823 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:38:01.904802    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3360b04a-23e1-49f9-84e4-11186ccde5e7-original-pull-secret podName:3360b04a-23e1-49f9-84e4-11186ccde5e7 nodeName:}" failed. No retries permitted until 2026-07-14 11:38:33.90478484 +0000 UTC m=+65.136594115 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3360b04a-23e1-49f9-84e4-11186ccde5e7-original-pull-secret") pod "global-pull-secret-syncer-75pwf" (UID: "3360b04a-23e1-49f9-84e4-11186ccde5e7") : object "kube-system"/"original-pull-secret" not registered
Jul 14 11:38:01.904999 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:38:01.904828    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5eafaa1-c985-4204-809a-3e9a52c96e55-metrics-certs podName:a5eafaa1-c985-4204-809a-3e9a52c96e55 nodeName:}" failed. No retries permitted until 2026-07-14 11:38:33.90481088 +0000 UTC m=+65.136620183 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a5eafaa1-c985-4204-809a-3e9a52c96e55-metrics-certs") pod "network-metrics-daemon-66zbn" (UID: "a5eafaa1-c985-4204-809a-3e9a52c96e55") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 11:38:02.005190 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:02.005154    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77qmm\" (UniqueName: \"kubernetes.io/projected/2f44b67b-26da-4ab3-9dfa-343ad6334e4f-kube-api-access-77qmm\") pod \"network-check-target-4zvrn\" (UID: \"2f44b67b-26da-4ab3-9dfa-343ad6334e4f\") " pod="openshift-network-diagnostics/network-check-target-4zvrn"
Jul 14 11:38:02.005379 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:38:02.005330    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 14 11:38:02.005379 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:38:02.005350    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 14 11:38:02.005379 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:38:02.005360    2551 projected.go:196] Error preparing data for projected volume kube-api-access-77qmm for pod openshift-network-diagnostics/network-check-target-4zvrn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 11:38:02.005511 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:38:02.005434    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2f44b67b-26da-4ab3-9dfa-343ad6334e4f-kube-api-access-77qmm podName:2f44b67b-26da-4ab3-9dfa-343ad6334e4f nodeName:}" failed. No retries permitted until 2026-07-14 11:38:34.005420069 +0000 UTC m=+65.237229363 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-77qmm" (UniqueName: "kubernetes.io/projected/2f44b67b-26da-4ab3-9dfa-343ad6334e4f-kube-api-access-77qmm") pod "network-check-target-4zvrn" (UID: "2f44b67b-26da-4ab3-9dfa-343ad6334e4f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 11:38:02.185360 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:02.185333    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4zvrn"
Jul 14 11:38:02.185512 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:38:02.185445    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4zvrn" podUID="2f44b67b-26da-4ab3-9dfa-343ad6334e4f"
Jul 14 11:38:02.185512 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:02.185481    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-66zbn"
Jul 14 11:38:02.185607 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:38:02.185538    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-66zbn" podUID="a5eafaa1-c985-4204-809a-3e9a52c96e55"
Jul 14 11:38:03.185256 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.185222    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-75pwf"
Jul 14 11:38:03.185874 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:38:03.185345    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-75pwf" podUID="3360b04a-23e1-49f9-84e4-11186ccde5e7"
Jul 14 11:38:03.596328 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.596243    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-173.ec2.internal" event="NodeReady"
Jul 14 11:38:03.596477 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.596375    2551 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 14 11:38:03.633803 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.633766    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d574df866-thdlk"]
Jul 14 11:38:03.638287 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.638258    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bb5d66b49-vmx85"]
Jul 14 11:38:03.638424 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.638377    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d574df866-thdlk"
Jul 14 11:38:03.640946 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.640920    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 14 11:38:03.641203 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.641156    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 14 11:38:03.641311 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.641249    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-d74bs\""
Jul 14 11:38:03.641394 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.641373    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 14 11:38:03.641462 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.641408    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 14 11:38:03.642533 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.642513    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-c9cf54fd5-7xknq"]
Jul 14 11:38:03.642831 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.642742    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bb5d66b49-vmx85"
Jul 14 11:38:03.645544 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.645518    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 14 11:38:03.645544 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.645527    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 14 11:38:03.645832 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.645809    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 14 11:38:03.645965 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.645810    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 14 11:38:03.646644 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.646626    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-698bdff498-sjflm"]
Jul 14 11:38:03.647205 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.646744    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-c9cf54fd5-7xknq"
Jul 14 11:38:03.649543 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.649450    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-zdzfm\""
Jul 14 11:38:03.649713 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.649677    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 14 11:38:03.649919 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.649899    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 14 11:38:03.650096 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.650079    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 14 11:38:03.651108 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.650857    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d574df866-thdlk"]
Jul 14 11:38:03.651108 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.650973    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-698bdff498-sjflm"
Jul 14 11:38:03.651431 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.651409    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-698bdff498-sjflm"]
Jul 14 11:38:03.654982 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.654963    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 14 11:38:03.655257 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.655238    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 14 11:38:03.656615 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.656570    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-x8b5w"]
Jul 14 11:38:03.660428 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.660399    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-5qmr8"]
Jul 14 11:38:03.660558 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.660544    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-x8b5w"
Jul 14 11:38:03.664136 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.663938    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-8ht88\""
Jul 14 11:38:03.664252 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.664163    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 14 11:38:03.664252 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.664199    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 14 11:38:03.664358 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.664263    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 14 11:38:03.666183 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.665971    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-c9cf54fd5-7xknq"]
Jul 14 11:38:03.666183 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.666086    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5qmr8"
Jul 14 11:38:03.666716 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.666692    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bb5d66b49-vmx85"]
Jul 14 11:38:03.672384 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.668603    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 14 11:38:03.672384 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.668702    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-9dtrp\""
Jul 14 11:38:03.672384 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.669382    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 14 11:38:03.674115 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.674085    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-x8b5w"]
Jul 14 11:38:03.675827 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.675805    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5qmr8"]
Jul 14 11:38:03.719626 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.719570    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-td7gt\" (UniqueName: \"kubernetes.io/projected/6329a619-10fc-475e-88bb-b0573546fbd5-kube-api-access-td7gt\") pod \"cluster-proxy-proxy-agent-6bb5d66b49-vmx85\" (UID: \"6329a619-10fc-475e-88bb-b0573546fbd5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bb5d66b49-vmx85"
Jul 14 11:38:03.719815 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.719634    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/0a0efcd7-960f-4f3e-b9e0-b7bd5f19a743-klusterlet-config\") pod \"klusterlet-addon-workmgr-698bdff498-sjflm\" (UID: \"0a0efcd7-960f-4f3e-b9e0-b7bd5f19a743\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-698bdff498-sjflm"
Jul 14 11:38:03.719815 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.719730    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/6329a619-10fc-475e-88bb-b0573546fbd5-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6bb5d66b49-vmx85\" (UID: \"6329a619-10fc-475e-88bb-b0573546fbd5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bb5d66b49-vmx85"
Jul 14 11:38:03.719815 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.719774    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/6329a619-10fc-475e-88bb-b0573546fbd5-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6bb5d66b49-vmx85\" (UID: \"6329a619-10fc-475e-88bb-b0573546fbd5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bb5d66b49-vmx85"
Jul 14 11:38:03.719815 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.719802    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/41b152d0-1171-4eb6-82eb-40e5d42a76b5-registry-tls\") pod \"image-registry-c9cf54fd5-7xknq\" (UID: \"41b152d0-1171-4eb6-82eb-40e5d42a76b5\") " pod="openshift-image-registry/image-registry-c9cf54fd5-7xknq"
Jul 14 11:38:03.719981 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.719880    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3c59746c-ccd0-441f-bea7-f22762395f47-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5d574df866-thdlk\" (UID: \"3c59746c-ccd0-441f-bea7-f22762395f47\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d574df866-thdlk"
Jul 14 11:38:03.719981 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.719920    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/41b152d0-1171-4eb6-82eb-40e5d42a76b5-ca-trust-extracted\") pod \"image-registry-c9cf54fd5-7xknq\" (UID: \"41b152d0-1171-4eb6-82eb-40e5d42a76b5\") " pod="openshift-image-registry/image-registry-c9cf54fd5-7xknq"
Jul 14 11:38:03.719981 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.719952    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0a0efcd7-960f-4f3e-b9e0-b7bd5f19a743-tmp\") pod \"klusterlet-addon-workmgr-698bdff498-sjflm\" (UID: \"0a0efcd7-960f-4f3e-b9e0-b7bd5f19a743\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-698bdff498-sjflm"
Jul 14 11:38:03.720069 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.719986    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbgh2\" (UniqueName: \"kubernetes.io/projected/3c59746c-ccd0-441f-bea7-f22762395f47-kube-api-access-cbgh2\") pod \"managed-serviceaccount-addon-agent-5d574df866-thdlk\" (UID: \"3c59746c-ccd0-441f-bea7-f22762395f47\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d574df866-thdlk"
Jul 14 11:38:03.720069 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.720012    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/41b152d0-1171-4eb6-82eb-40e5d42a76b5-registry-certificates\") pod \"image-registry-c9cf54fd5-7xknq\" (UID: \"41b152d0-1171-4eb6-82eb-40e5d42a76b5\") " pod="openshift-image-registry/image-registry-c9cf54fd5-7xknq"
Jul 14 11:38:03.720069 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.720033    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/41b152d0-1171-4eb6-82eb-40e5d42a76b5-bound-sa-token\") pod \"image-registry-c9cf54fd5-7xknq\" (UID: \"41b152d0-1171-4eb6-82eb-40e5d42a76b5\") " pod="openshift-image-registry/image-registry-c9cf54fd5-7xknq"
Jul 14 11:38:03.720069 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.720055    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6dfd\" (UniqueName: \"kubernetes.io/projected/0a0efcd7-960f-4f3e-b9e0-b7bd5f19a743-kube-api-access-x6dfd\") pod \"klusterlet-addon-workmgr-698bdff498-sjflm\" (UID: \"0a0efcd7-960f-4f3e-b9e0-b7bd5f19a743\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-698bdff498-sjflm"
Jul 14 11:38:03.720200 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.720108    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/6329a619-10fc-475e-88bb-b0573546fbd5-ca\") pod \"cluster-proxy-proxy-agent-6bb5d66b49-vmx85\" (UID: \"6329a619-10fc-475e-88bb-b0573546fbd5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bb5d66b49-vmx85"
Jul 14 11:38:03.720200 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.720138    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6329a619-10fc-475e-88bb-b0573546fbd5-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6bb5d66b49-vmx85\" (UID: \"6329a619-10fc-475e-88bb-b0573546fbd5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bb5d66b49-vmx85"
Jul 14 11:38:03.720200 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.720172    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/41b152d0-1171-4eb6-82eb-40e5d42a76b5-image-registry-private-configuration\") pod \"image-registry-c9cf54fd5-7xknq\" (UID: \"41b152d0-1171-4eb6-82eb-40e5d42a76b5\") " pod="openshift-image-registry/image-registry-c9cf54fd5-7xknq"
Jul 14 11:38:03.720302 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.720203    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/41b152d0-1171-4eb6-82eb-40e5d42a76b5-trusted-ca\") pod \"image-registry-c9cf54fd5-7xknq\" (UID: \"41b152d0-1171-4eb6-82eb-40e5d42a76b5\") " pod="openshift-image-registry/image-registry-c9cf54fd5-7xknq"
Jul 14 11:38:03.720302 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.720219    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/6329a619-10fc-475e-88bb-b0573546fbd5-hub\") pod \"cluster-proxy-proxy-agent-6bb5d66b49-vmx85\" (UID: \"6329a619-10fc-475e-88bb-b0573546fbd5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bb5d66b49-vmx85"
Jul 14 11:38:03.720302 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.720235    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/41b152d0-1171-4eb6-82eb-40e5d42a76b5-installation-pull-secrets\") pod \"image-registry-c9cf54fd5-7xknq\" (UID: \"41b152d0-1171-4eb6-82eb-40e5d42a76b5\") " pod="openshift-image-registry/image-registry-c9cf54fd5-7xknq"
Jul 14 11:38:03.720302 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.720253    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdwn4\" (UniqueName: \"kubernetes.io/projected/41b152d0-1171-4eb6-82eb-40e5d42a76b5-kube-api-access-fdwn4\") pod \"image-registry-c9cf54fd5-7xknq\" (UID: \"41b152d0-1171-4eb6-82eb-40e5d42a76b5\") " pod="openshift-image-registry/image-registry-c9cf54fd5-7xknq"
Jul 14 11:38:03.820812 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.820777    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-td7gt\" (UniqueName: \"kubernetes.io/projected/6329a619-10fc-475e-88bb-b0573546fbd5-kube-api-access-td7gt\") pod \"cluster-proxy-proxy-agent-6bb5d66b49-vmx85\" (UID: \"6329a619-10fc-475e-88bb-b0573546fbd5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bb5d66b49-vmx85"
Jul 14 11:38:03.820812 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.820814    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/0a0efcd7-960f-4f3e-b9e0-b7bd5f19a743-klusterlet-config\") pod \"klusterlet-addon-workmgr-698bdff498-sjflm\" (UID: \"0a0efcd7-960f-4f3e-b9e0-b7bd5f19a743\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-698bdff498-sjflm"
Jul 14 11:38:03.821060 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.820839    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4f703685-4285-43da-a074-362a513a32b1-config-volume\") pod \"dns-default-5qmr8\" (UID: \"4f703685-4285-43da-a074-362a513a32b1\") " pod="openshift-dns/dns-default-5qmr8"
Jul 14 11:38:03.821060 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.820888    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/6329a619-10fc-475e-88bb-b0573546fbd5-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6bb5d66b49-vmx85\" (UID: \"6329a619-10fc-475e-88bb-b0573546fbd5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bb5d66b49-vmx85"
Jul 14 11:38:03.821060 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.820918    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/6329a619-10fc-475e-88bb-b0573546fbd5-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6bb5d66b49-vmx85\" (UID: \"6329a619-10fc-475e-88bb-b0573546fbd5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bb5d66b49-vmx85"
Jul 14 11:38:03.821060 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.820945    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/41b152d0-1171-4eb6-82eb-40e5d42a76b5-registry-tls\") pod \"image-registry-c9cf54fd5-7xknq\" (UID: \"41b152d0-1171-4eb6-82eb-40e5d42a76b5\") " pod="openshift-image-registry/image-registry-c9cf54fd5-7xknq"
Jul 14 11:38:03.821060 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.820981    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3c59746c-ccd0-441f-bea7-f22762395f47-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5d574df866-thdlk\" (UID: \"3c59746c-ccd0-441f-bea7-f22762395f47\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d574df866-thdlk"
Jul 14 11:38:03.821060 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.821007    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/41b152d0-1171-4eb6-82eb-40e5d42a76b5-ca-trust-extracted\") pod \"image-registry-c9cf54fd5-7xknq\" (UID: \"41b152d0-1171-4eb6-82eb-40e5d42a76b5\") " pod="openshift-image-registry/image-registry-c9cf54fd5-7xknq"
Jul 14 11:38:03.821060 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.821033    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0a0efcd7-960f-4f3e-b9e0-b7bd5f19a743-tmp\") pod \"klusterlet-addon-workmgr-698bdff498-sjflm\" (UID: \"0a0efcd7-960f-4f3e-b9e0-b7bd5f19a743\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-698bdff498-sjflm"
Jul 14 11:38:03.821383 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:38:03.821123    2551 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 14 11:38:03.821383 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:38:03.821146    2551 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-c9cf54fd5-7xknq: secret "image-registry-tls" not found
Jul 14 11:38:03.821383 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:38:03.821206    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/41b152d0-1171-4eb6-82eb-40e5d42a76b5-registry-tls podName:41b152d0-1171-4eb6-82eb-40e5d42a76b5 nodeName:}" failed. No retries permitted until 2026-07-14 11:38:04.321185378 +0000 UTC m=+35.552994675 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/41b152d0-1171-4eb6-82eb-40e5d42a76b5-registry-tls") pod "image-registry-c9cf54fd5-7xknq" (UID: "41b152d0-1171-4eb6-82eb-40e5d42a76b5") : secret "image-registry-tls" not found
Jul 14 11:38:03.821537 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.821448    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f703685-4285-43da-a074-362a513a32b1-tmp-dir\") pod \"dns-default-5qmr8\" (UID: \"4f703685-4285-43da-a074-362a513a32b1\") " pod="openshift-dns/dns-default-5qmr8"
Jul 14 11:38:03.821537 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.821473    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/41b152d0-1171-4eb6-82eb-40e5d42a76b5-ca-trust-extracted\") pod \"image-registry-c9cf54fd5-7xknq\" (UID: \"41b152d0-1171-4eb6-82eb-40e5d42a76b5\") " pod="openshift-image-registry/image-registry-c9cf54fd5-7xknq"
Jul 14 11:38:03.821537 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.821491    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbgh2\" (UniqueName: \"kubernetes.io/projected/3c59746c-ccd0-441f-bea7-f22762395f47-kube-api-access-cbgh2\") pod \"managed-serviceaccount-addon-agent-5d574df866-thdlk\" (UID: \"3c59746c-ccd0-441f-bea7-f22762395f47\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d574df866-thdlk"
Jul 14 11:38:03.821710 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.821521    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/41b152d0-1171-4eb6-82eb-40e5d42a76b5-registry-certificates\") pod \"image-registry-c9cf54fd5-7xknq\" (UID: \"41b152d0-1171-4eb6-82eb-40e5d42a76b5\") " pod="openshift-image-registry/image-registry-c9cf54fd5-7xknq"
Jul 14 11:38:03.821710 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.821603    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/41b152d0-1171-4eb6-82eb-40e5d42a76b5-bound-sa-token\") pod \"image-registry-c9cf54fd5-7xknq\" (UID: \"41b152d0-1171-4eb6-82eb-40e5d42a76b5\") " pod="openshift-image-registry/image-registry-c9cf54fd5-7xknq"
Jul 14 11:38:03.821710 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.821628    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6dfd\" (UniqueName: \"kubernetes.io/projected/0a0efcd7-960f-4f3e-b9e0-b7bd5f19a743-kube-api-access-x6dfd\") pod \"klusterlet-addon-workmgr-698bdff498-sjflm\" (UID: \"0a0efcd7-960f-4f3e-b9e0-b7bd5f19a743\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-698bdff498-sjflm"
Jul 14 11:38:03.821710 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.821678    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zh5mn\" (UniqueName: \"kubernetes.io/projected/4f703685-4285-43da-a074-362a513a32b1-kube-api-access-zh5mn\") pod \"dns-default-5qmr8\" (UID: \"4f703685-4285-43da-a074-362a513a32b1\") " pod="openshift-dns/dns-default-5qmr8"
Jul 14 11:38:03.821710 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.821699    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/6329a619-10fc-475e-88bb-b0573546fbd5-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6bb5d66b49-vmx85\" (UID: \"6329a619-10fc-475e-88bb-b0573546fbd5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bb5d66b49-vmx85"
Jul 14 11:38:03.821945 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.821721    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4f703685-4285-43da-a074-362a513a32b1-metrics-tls\") pod \"dns-default-5qmr8\" (UID: \"4f703685-4285-43da-a074-362a513a32b1\") " pod="openshift-dns/dns-default-5qmr8"
Jul 14 11:38:03.821945 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.821752    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/6329a619-10fc-475e-88bb-b0573546fbd5-ca\") pod \"cluster-proxy-proxy-agent-6bb5d66b49-vmx85\" (UID: \"6329a619-10fc-475e-88bb-b0573546fbd5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bb5d66b49-vmx85"
Jul 14 11:38:03.821945 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.821776    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6329a619-10fc-475e-88bb-b0573546fbd5-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6bb5d66b49-vmx85\" (UID: \"6329a619-10fc-475e-88bb-b0573546fbd5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bb5d66b49-vmx85"
Jul 14 11:38:03.821945 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.821822    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/41b152d0-1171-4eb6-82eb-40e5d42a76b5-image-registry-private-configuration\") pod \"image-registry-c9cf54fd5-7xknq\" (UID: \"41b152d0-1171-4eb6-82eb-40e5d42a76b5\") " pod="openshift-image-registry/image-registry-c9cf54fd5-7xknq"
Jul 14 11:38:03.821945 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.821848    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/41b152d0-1171-4eb6-82eb-40e5d42a76b5-trusted-ca\") pod \"image-registry-c9cf54fd5-7xknq\" (UID: \"41b152d0-1171-4eb6-82eb-40e5d42a76b5\") " pod="openshift-image-registry/image-registry-c9cf54fd5-7xknq"
Jul 14 11:38:03.821945 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.821869    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/6329a619-10fc-475e-88bb-b0573546fbd5-hub\") pod \"cluster-proxy-proxy-agent-6bb5d66b49-vmx85\" (UID: \"6329a619-10fc-475e-88bb-b0573546fbd5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bb5d66b49-vmx85"
Jul 14 11:38:03.821945 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.821910    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/41b152d0-1171-4eb6-82eb-40e5d42a76b5-installation-pull-secrets\") pod \"image-registry-c9cf54fd5-7xknq\" (UID: \"41b152d0-1171-4eb6-82eb-40e5d42a76b5\") " pod="openshift-image-registry/image-registry-c9cf54fd5-7xknq"
Jul 14 11:38:03.821945 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.821932    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdwn4\" (UniqueName: \"kubernetes.io/projected/41b152d0-1171-4eb6-82eb-40e5d42a76b5-kube-api-access-fdwn4\") pod \"image-registry-c9cf54fd5-7xknq\" (UID: \"41b152d0-1171-4eb6-82eb-40e5d42a76b5\") " pod="openshift-image-registry/image-registry-c9cf54fd5-7xknq"
Jul 14 11:38:03.822379 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.821958    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2585982a-d4d0-4d80-9b85-fbfe27886ab2-cert\") pod \"ingress-canary-x8b5w\" (UID: \"2585982a-d4d0-4d80-9b85-fbfe27886ab2\") " pod="openshift-ingress-canary/ingress-canary-x8b5w"
Jul 14 11:38:03.822379 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.821979    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9c29j\" (UniqueName: \"kubernetes.io/projected/2585982a-d4d0-4d80-9b85-fbfe27886ab2-kube-api-access-9c29j\") pod \"ingress-canary-x8b5w\" (UID: \"2585982a-d4d0-4d80-9b85-fbfe27886ab2\") " pod="openshift-ingress-canary/ingress-canary-x8b5w"
Jul 14 11:38:03.822379 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.822119    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/41b152d0-1171-4eb6-82eb-40e5d42a76b5-registry-certificates\") pod \"image-registry-c9cf54fd5-7xknq\" (UID: \"41b152d0-1171-4eb6-82eb-40e5d42a76b5\") " pod="openshift-image-registry/image-registry-c9cf54fd5-7xknq"
Jul 14 11:38:03.823073 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.823049    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/41b152d0-1171-4eb6-82eb-40e5d42a76b5-trusted-ca\") pod \"image-registry-c9cf54fd5-7xknq\" (UID: \"41b152d0-1171-4eb6-82eb-40e5d42a76b5\") " pod="openshift-image-registry/image-registry-c9cf54fd5-7xknq"
Jul 14 11:38:03.823556 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.823530    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0a0efcd7-960f-4f3e-b9e0-b7bd5f19a743-tmp\") pod \"klusterlet-addon-workmgr-698bdff498-sjflm\" (UID: \"0a0efcd7-960f-4f3e-b9e0-b7bd5f19a743\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-698bdff498-sjflm"
Jul 14 11:38:03.826500 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.826468    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/6329a619-10fc-475e-88bb-b0573546fbd5-hub\") pod \"cluster-proxy-proxy-agent-6bb5d66b49-vmx85\" (UID: \"6329a619-10fc-475e-88bb-b0573546fbd5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bb5d66b49-vmx85"
Jul 14 11:38:03.826500 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.826485    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6329a619-10fc-475e-88bb-b0573546fbd5-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6bb5d66b49-vmx85\" (UID: \"6329a619-10fc-475e-88bb-b0573546fbd5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bb5d66b49-vmx85"
Jul 14 11:38:03.826728 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.826540    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/0a0efcd7-960f-4f3e-b9e0-b7bd5f19a743-klusterlet-config\") pod \"klusterlet-addon-workmgr-698bdff498-sjflm\" (UID: \"0a0efcd7-960f-4f3e-b9e0-b7bd5f19a743\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-698bdff498-sjflm"
Jul 14 11:38:03.826728 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.826544    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3c59746c-ccd0-441f-bea7-f22762395f47-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5d574df866-thdlk\" (UID: \"3c59746c-ccd0-441f-bea7-f22762395f47\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d574df866-thdlk"
Jul 14 11:38:03.826977 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.826929    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/41b152d0-1171-4eb6-82eb-40e5d42a76b5-installation-pull-secrets\") pod \"image-registry-c9cf54fd5-7xknq\" (UID: \"41b152d0-1171-4eb6-82eb-40e5d42a76b5\") " pod="openshift-image-registry/image-registry-c9cf54fd5-7xknq"
Jul 14 11:38:03.827101 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.827078    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/6329a619-10fc-475e-88bb-b0573546fbd5-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6bb5d66b49-vmx85\" (UID: \"6329a619-10fc-475e-88bb-b0573546fbd5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bb5d66b49-vmx85"
Jul 14 11:38:03.827442 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.827425    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/41b152d0-1171-4eb6-82eb-40e5d42a76b5-image-registry-private-configuration\") pod \"image-registry-c9cf54fd5-7xknq\" (UID: \"41b152d0-1171-4eb6-82eb-40e5d42a76b5\") " pod="openshift-image-registry/image-registry-c9cf54fd5-7xknq"
Jul 14 11:38:03.827519 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.827493    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/6329a619-10fc-475e-88bb-b0573546fbd5-ca\") pod \"cluster-proxy-proxy-agent-6bb5d66b49-vmx85\" (UID: \"6329a619-10fc-475e-88bb-b0573546fbd5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bb5d66b49-vmx85"
Jul 14 11:38:03.827952 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.827927    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:38:03.831295 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.831260    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbgh2\" (UniqueName: \"kubernetes.io/projected/3c59746c-ccd0-441f-bea7-f22762395f47-kube-api-access-cbgh2\") pod \"managed-serviceaccount-addon-agent-5d574df866-thdlk\" (UID: \"3c59746c-ccd0-441f-bea7-f22762395f47\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d574df866-thdlk"
Jul 14 11:38:03.831851 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.831824    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/41b152d0-1171-4eb6-82eb-40e5d42a76b5-bound-sa-token\") pod \"image-registry-c9cf54fd5-7xknq\" (UID: \"41b152d0-1171-4eb6-82eb-40e5d42a76b5\") " pod="openshift-image-registry/image-registry-c9cf54fd5-7xknq"
Jul 14 11:38:03.832011 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.831989    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-td7gt\" (UniqueName: \"kubernetes.io/projected/6329a619-10fc-475e-88bb-b0573546fbd5-kube-api-access-td7gt\") pod \"cluster-proxy-proxy-agent-6bb5d66b49-vmx85\" (UID: \"6329a619-10fc-475e-88bb-b0573546fbd5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bb5d66b49-vmx85"
Jul 14 11:38:03.832084 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.832014    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6dfd\" (UniqueName: \"kubernetes.io/projected/0a0efcd7-960f-4f3e-b9e0-b7bd5f19a743-kube-api-access-x6dfd\") pod \"klusterlet-addon-workmgr-698bdff498-sjflm\" (UID: \"0a0efcd7-960f-4f3e-b9e0-b7bd5f19a743\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-698bdff498-sjflm"
Jul 14 11:38:03.832649 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.832565    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdwn4\" (UniqueName: \"kubernetes.io/projected/41b152d0-1171-4eb6-82eb-40e5d42a76b5-kube-api-access-fdwn4\") pod \"image-registry-c9cf54fd5-7xknq\" (UID: \"41b152d0-1171-4eb6-82eb-40e5d42a76b5\") " pod="openshift-image-registry/image-registry-c9cf54fd5-7xknq"
Jul 14 11:38:03.922349 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.922270    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2585982a-d4d0-4d80-9b85-fbfe27886ab2-cert\") pod \"ingress-canary-x8b5w\" (UID: \"2585982a-d4d0-4d80-9b85-fbfe27886ab2\") " pod="openshift-ingress-canary/ingress-canary-x8b5w"
Jul 14 11:38:03.922349 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.922308    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9c29j\" (UniqueName: \"kubernetes.io/projected/2585982a-d4d0-4d80-9b85-fbfe27886ab2-kube-api-access-9c29j\") pod \"ingress-canary-x8b5w\" (UID: \"2585982a-d4d0-4d80-9b85-fbfe27886ab2\") " pod="openshift-ingress-canary/ingress-canary-x8b5w"
Jul 14 11:38:03.922349 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.922329    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4f703685-4285-43da-a074-362a513a32b1-config-volume\") pod \"dns-default-5qmr8\" (UID: \"4f703685-4285-43da-a074-362a513a32b1\") " pod="openshift-dns/dns-default-5qmr8"
Jul 14 11:38:03.922610 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.922403    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f703685-4285-43da-a074-362a513a32b1-tmp-dir\") pod \"dns-default-5qmr8\" (UID: \"4f703685-4285-43da-a074-362a513a32b1\") " pod="openshift-dns/dns-default-5qmr8"
Jul 14 11:38:03.922610 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.922445    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh5mn\" (UniqueName: \"kubernetes.io/projected/4f703685-4285-43da-a074-362a513a32b1-kube-api-access-zh5mn\") pod \"dns-default-5qmr8\" (UID: \"4f703685-4285-43da-a074-362a513a32b1\") " pod="openshift-dns/dns-default-5qmr8"
Jul 14 11:38:03.922737 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.922712    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4f703685-4285-43da-a074-362a513a32b1-metrics-tls\") pod \"dns-default-5qmr8\" (UID: \"4f703685-4285-43da-a074-362a513a32b1\") " pod="openshift-dns/dns-default-5qmr8"
Jul 14 11:38:03.922804 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:38:03.922783    2551 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 14 11:38:03.922856 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.922826    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f703685-4285-43da-a074-362a513a32b1-tmp-dir\") pod \"dns-default-5qmr8\" (UID: \"4f703685-4285-43da-a074-362a513a32b1\") " pod="openshift-dns/dns-default-5qmr8"
Jul 14 11:38:03.922898 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:38:03.922866    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f703685-4285-43da-a074-362a513a32b1-metrics-tls podName:4f703685-4285-43da-a074-362a513a32b1 nodeName:}" failed. No retries permitted until 2026-07-14 11:38:04.422846241 +0000 UTC m=+35.654655527 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4f703685-4285-43da-a074-362a513a32b1-metrics-tls") pod "dns-default-5qmr8" (UID: "4f703685-4285-43da-a074-362a513a32b1") : secret "dns-default-metrics-tls" not found
Jul 14 11:38:03.923075 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.923042    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4f703685-4285-43da-a074-362a513a32b1-config-volume\") pod \"dns-default-5qmr8\" (UID: \"4f703685-4285-43da-a074-362a513a32b1\") " pod="openshift-dns/dns-default-5qmr8"
Jul 14 11:38:03.925193 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.925166    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2585982a-d4d0-4d80-9b85-fbfe27886ab2-cert\") pod \"ingress-canary-x8b5w\" (UID: \"2585982a-d4d0-4d80-9b85-fbfe27886ab2\") " pod="openshift-ingress-canary/ingress-canary-x8b5w"
Jul 14 11:38:03.933524 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.933491    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zh5mn\" (UniqueName: \"kubernetes.io/projected/4f703685-4285-43da-a074-362a513a32b1-kube-api-access-zh5mn\") pod \"dns-default-5qmr8\" (UID: \"4f703685-4285-43da-a074-362a513a32b1\") " pod="openshift-dns/dns-default-5qmr8"
Jul 14 11:38:03.933747 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.933726    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c29j\" (UniqueName: \"kubernetes.io/projected/2585982a-d4d0-4d80-9b85-fbfe27886ab2-kube-api-access-9c29j\") pod \"ingress-canary-x8b5w\" (UID: \"2585982a-d4d0-4d80-9b85-fbfe27886ab2\") " pod="openshift-ingress-canary/ingress-canary-x8b5w"
Jul 14 11:38:03.949471 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.949428    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:38:03.959288 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.959255    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d574df866-thdlk"
Jul 14 11:38:03.970704 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.970674    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bb5d66b49-vmx85"
Jul 14 11:38:03.995437 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:03.995403    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-698bdff498-sjflm"
Jul 14 11:38:04.002922 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:04.002892    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-x8b5w"
Jul 14 11:38:04.185493 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:04.185464    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4zvrn"
Jul 14 11:38:04.186276 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:04.185699    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-66zbn"
Jul 14 11:38:04.188630 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:04.188609    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 14 11:38:04.188757 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:04.188637    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-9256t\""
Jul 14 11:38:04.188757 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:04.188656    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 14 11:38:04.188864 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:04.188803    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 14 11:38:04.188864 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:04.188806    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-26j7k\""
Jul 14 11:38:04.325466 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:04.325423    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/41b152d0-1171-4eb6-82eb-40e5d42a76b5-registry-tls\") pod \"image-registry-c9cf54fd5-7xknq\" (UID: \"41b152d0-1171-4eb6-82eb-40e5d42a76b5\") " pod="openshift-image-registry/image-registry-c9cf54fd5-7xknq"
Jul 14 11:38:04.325658 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:38:04.325603    2551 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 14 11:38:04.325658 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:38:04.325620    2551 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-c9cf54fd5-7xknq: secret "image-registry-tls" not found
Jul 14 11:38:04.325761 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:38:04.325688    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/41b152d0-1171-4eb6-82eb-40e5d42a76b5-registry-tls podName:41b152d0-1171-4eb6-82eb-40e5d42a76b5 nodeName:}" failed. No retries permitted until 2026-07-14 11:38:05.325667328 +0000 UTC m=+36.557476605 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/41b152d0-1171-4eb6-82eb-40e5d42a76b5-registry-tls") pod "image-registry-c9cf54fd5-7xknq" (UID: "41b152d0-1171-4eb6-82eb-40e5d42a76b5") : secret "image-registry-tls" not found
Jul 14 11:38:04.426462 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:04.426426    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4f703685-4285-43da-a074-362a513a32b1-metrics-tls\") pod \"dns-default-5qmr8\" (UID: \"4f703685-4285-43da-a074-362a513a32b1\") " pod="openshift-dns/dns-default-5qmr8"
Jul 14 11:38:04.426659 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:38:04.426554    2551 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 14 11:38:04.426659 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:38:04.426629    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f703685-4285-43da-a074-362a513a32b1-metrics-tls podName:4f703685-4285-43da-a074-362a513a32b1 nodeName:}" failed. No retries permitted until 2026-07-14 11:38:05.42661093 +0000 UTC m=+36.658420209 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4f703685-4285-43da-a074-362a513a32b1-metrics-tls") pod "dns-default-5qmr8" (UID: "4f703685-4285-43da-a074-362a513a32b1") : secret "dns-default-metrics-tls" not found
Jul 14 11:38:05.185127 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:05.185086    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-75pwf"
Jul 14 11:38:05.188442 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:05.188420    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 14 11:38:05.336263 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:05.336223    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/41b152d0-1171-4eb6-82eb-40e5d42a76b5-registry-tls\") pod \"image-registry-c9cf54fd5-7xknq\" (UID: \"41b152d0-1171-4eb6-82eb-40e5d42a76b5\") " pod="openshift-image-registry/image-registry-c9cf54fd5-7xknq"
Jul 14 11:38:05.336452 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:38:05.336369    2551 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 14 11:38:05.336452 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:38:05.336391    2551 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-c9cf54fd5-7xknq: secret "image-registry-tls" not found
Jul 14 11:38:05.336557 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:38:05.336457    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/41b152d0-1171-4eb6-82eb-40e5d42a76b5-registry-tls podName:41b152d0-1171-4eb6-82eb-40e5d42a76b5 nodeName:}" failed. No retries permitted until 2026-07-14 11:38:07.33643537 +0000 UTC m=+38.568244646 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/41b152d0-1171-4eb6-82eb-40e5d42a76b5-registry-tls") pod "image-registry-c9cf54fd5-7xknq" (UID: "41b152d0-1171-4eb6-82eb-40e5d42a76b5") : secret "image-registry-tls" not found
Jul 14 11:38:05.437458 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:05.437387    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4f703685-4285-43da-a074-362a513a32b1-metrics-tls\") pod \"dns-default-5qmr8\" (UID: \"4f703685-4285-43da-a074-362a513a32b1\") " pod="openshift-dns/dns-default-5qmr8"
Jul 14 11:38:05.437611 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:38:05.437533    2551 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 14 11:38:05.437661 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:38:05.437615    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f703685-4285-43da-a074-362a513a32b1-metrics-tls podName:4f703685-4285-43da-a074-362a513a32b1 nodeName:}" failed. No retries permitted until 2026-07-14 11:38:07.437591432 +0000 UTC m=+38.669400723 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4f703685-4285-43da-a074-362a513a32b1-metrics-tls") pod "dns-default-5qmr8" (UID: "4f703685-4285-43da-a074-362a513a32b1") : secret "dns-default-metrics-tls" not found
Jul 14 11:38:05.636493 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:05.636330    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-x8b5w"]
Jul 14 11:38:05.645933 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:05.645905    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d574df866-thdlk"]
Jul 14 11:38:05.646975 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:05.646951    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bb5d66b49-vmx85"]
Jul 14 11:38:05.650559 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:38:05.650334    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6329a619_10fc_475e_88bb_b0573546fbd5.slice/crio-4cdc6d94c213d460e7b01e29e6154934ba7e79731418b8dad78aeb88f68ec76f WatchSource:0}: Error finding container 4cdc6d94c213d460e7b01e29e6154934ba7e79731418b8dad78aeb88f68ec76f: Status 404 returned error can't find the container with id 4cdc6d94c213d460e7b01e29e6154934ba7e79731418b8dad78aeb88f68ec76f
Jul 14 11:38:05.652184 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:05.652051    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-698bdff498-sjflm"]
Jul 14 11:38:05.657499 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:38:05.657444    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a0efcd7_960f_4f3e_b9e0_b7bd5f19a743.slice/crio-f7af2d8aacbba69f4c2edd1b83f38addcf9994fe5c1b9e609fd3b048009ef41d WatchSource:0}: Error finding container f7af2d8aacbba69f4c2edd1b83f38addcf9994fe5c1b9e609fd3b048009ef41d: Status 404 returned error can't find the container with id f7af2d8aacbba69f4c2edd1b83f38addcf9994fe5c1b9e609fd3b048009ef41d
Jul 14 11:38:06.445526 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:06.445476    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-x8b5w" event={"ID":"2585982a-d4d0-4d80-9b85-fbfe27886ab2","Type":"ContainerStarted","Data":"2c880b5ccd062ba7912793eee3eb56f0fa12b6f2cdce856e2441efeaf8c37f62"}
Jul 14 11:38:06.446733 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:06.446702    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d574df866-thdlk" event={"ID":"3c59746c-ccd0-441f-bea7-f22762395f47","Type":"ContainerStarted","Data":"598b61cb6b6eeaaf02d44cbec8dd25c381356a052ab37d8ac02696e82deaf1d8"}
Jul 14 11:38:06.448190 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:06.448127    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-698bdff498-sjflm" event={"ID":"0a0efcd7-960f-4f3e-b9e0-b7bd5f19a743","Type":"ContainerStarted","Data":"f7af2d8aacbba69f4c2edd1b83f38addcf9994fe5c1b9e609fd3b048009ef41d"}
Jul 14 11:38:06.453141 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:06.453115    2551 generic.go:356] "Generic (PLEG): container finished" podID="4c7434bb-d4dd-407e-a009-33d672d4bfc6" containerID="3b3881c47c84aebb03389867583e09e371aa93d1e00c80aa47779f693c60a7c8" exitCode=0
Jul 14 11:38:06.453265 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:06.453210    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4fhlr" event={"ID":"4c7434bb-d4dd-407e-a009-33d672d4bfc6","Type":"ContainerDied","Data":"3b3881c47c84aebb03389867583e09e371aa93d1e00c80aa47779f693c60a7c8"}
Jul 14 11:38:06.454447 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:06.454423    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bb5d66b49-vmx85" event={"ID":"6329a619-10fc-475e-88bb-b0573546fbd5","Type":"ContainerStarted","Data":"4cdc6d94c213d460e7b01e29e6154934ba7e79731418b8dad78aeb88f68ec76f"}
Jul 14 11:38:07.356437 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:07.356398    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/41b152d0-1171-4eb6-82eb-40e5d42a76b5-registry-tls\") pod \"image-registry-c9cf54fd5-7xknq\" (UID: \"41b152d0-1171-4eb6-82eb-40e5d42a76b5\") " pod="openshift-image-registry/image-registry-c9cf54fd5-7xknq"
Jul 14 11:38:07.356645 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:38:07.356609    2551 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 14 11:38:07.356645 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:38:07.356635    2551 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-c9cf54fd5-7xknq: secret "image-registry-tls" not found
Jul 14 11:38:07.356788 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:38:07.356707    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/41b152d0-1171-4eb6-82eb-40e5d42a76b5-registry-tls podName:41b152d0-1171-4eb6-82eb-40e5d42a76b5 nodeName:}" failed. No retries permitted until 2026-07-14 11:38:11.356684585 +0000 UTC m=+42.588493862 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/41b152d0-1171-4eb6-82eb-40e5d42a76b5-registry-tls") pod "image-registry-c9cf54fd5-7xknq" (UID: "41b152d0-1171-4eb6-82eb-40e5d42a76b5") : secret "image-registry-tls" not found
Jul 14 11:38:07.456960 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:07.456897    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4f703685-4285-43da-a074-362a513a32b1-metrics-tls\") pod \"dns-default-5qmr8\" (UID: \"4f703685-4285-43da-a074-362a513a32b1\") " pod="openshift-dns/dns-default-5qmr8"
Jul 14 11:38:07.457513 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:38:07.457052    2551 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 14 11:38:07.457513 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:38:07.457116    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f703685-4285-43da-a074-362a513a32b1-metrics-tls podName:4f703685-4285-43da-a074-362a513a32b1 nodeName:}" failed. No retries permitted until 2026-07-14 11:38:11.457095307 +0000 UTC m=+42.688904600 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4f703685-4285-43da-a074-362a513a32b1-metrics-tls") pod "dns-default-5qmr8" (UID: "4f703685-4285-43da-a074-362a513a32b1") : secret "dns-default-metrics-tls" not found
Jul 14 11:38:07.461857 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:07.461746    2551 generic.go:356] "Generic (PLEG): container finished" podID="4c7434bb-d4dd-407e-a009-33d672d4bfc6" containerID="a7cfd907674213b58fd5e5455921d036f9639ad7361e75d8736e6f337713bf77" exitCode=0
Jul 14 11:38:07.461857 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:07.461803    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4fhlr" event={"ID":"4c7434bb-d4dd-407e-a009-33d672d4bfc6","Type":"ContainerDied","Data":"a7cfd907674213b58fd5e5455921d036f9639ad7361e75d8736e6f337713bf77"}
Jul 14 11:38:08.541675 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:08.541634    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:38:09.981151 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:09.981107    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:38:10.472640 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:10.472601    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-x8b5w" event={"ID":"2585982a-d4d0-4d80-9b85-fbfe27886ab2","Type":"ContainerStarted","Data":"da4fa7615ecad8f05fafb905f8bbba94f89e1b4bbd7c4a2e92aa7a46c630b9bf"}
Jul 14 11:38:10.476191 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:10.476163    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4fhlr" event={"ID":"4c7434bb-d4dd-407e-a009-33d672d4bfc6","Type":"ContainerStarted","Data":"f2f40dbcd4254bc70e8d06318207964d76b2f1c23a5eefac63f2c02a661b3632"}
Jul 14 11:38:10.510343 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:10.510287    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-4fhlr" podStartSLOduration=9.328566013 podStartE2EDuration="41.510271343s" totalImagesPullingTime="32.18170533s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 11:37:29 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 11:38:10.508789324 +0000 UTC m=+41.740598645" watchObservedRunningTime="2026-07-14 11:38:10.510271343 +0000 UTC m=+41.742080616"
Jul 14 11:38:11.115748 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:11.115712    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:38:11.257897 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:11.257865    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:38:11.357813 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:11.357708    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:38:11.389748 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:11.389714    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/41b152d0-1171-4eb6-82eb-40e5d42a76b5-registry-tls\") pod \"image-registry-c9cf54fd5-7xknq\" (UID: \"41b152d0-1171-4eb6-82eb-40e5d42a76b5\") " pod="openshift-image-registry/image-registry-c9cf54fd5-7xknq"
Jul 14 11:38:11.389881 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:38:11.389851    2551 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 14 11:38:11.389881 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:38:11.389863    2551 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-c9cf54fd5-7xknq: secret "image-registry-tls" not found
Jul 14 11:38:11.389958 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:38:11.389916    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/41b152d0-1171-4eb6-82eb-40e5d42a76b5-registry-tls podName:41b152d0-1171-4eb6-82eb-40e5d42a76b5 nodeName:}" failed. No retries permitted until 2026-07-14 11:38:19.389900034 +0000 UTC m=+50.621709314 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/41b152d0-1171-4eb6-82eb-40e5d42a76b5-registry-tls") pod "image-registry-c9cf54fd5-7xknq" (UID: "41b152d0-1171-4eb6-82eb-40e5d42a76b5") : secret "image-registry-tls" not found
Jul 14 11:38:11.481065 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:11.481022    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bb5d66b49-vmx85" event={"ID":"6329a619-10fc-475e-88bb-b0573546fbd5","Type":"ContainerStarted","Data":"b51d86b6c02671183dd10537e30cb824093ca331cddae7a769debbede03f33bb"}
Jul 14 11:38:11.481196 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:11.481076    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bb5d66b49-vmx85" event={"ID":"6329a619-10fc-475e-88bb-b0573546fbd5","Type":"ContainerStarted","Data":"ff4af5d329c1279c68487b241aebcf5e3ac91ee4a1295b266e9dc6ed4aa00c4d"}
Jul 14 11:38:11.481196 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:11.481091    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bb5d66b49-vmx85" event={"ID":"6329a619-10fc-475e-88bb-b0573546fbd5","Type":"ContainerStarted","Data":"6f11364d84534cda4a39472b54d224c2235d666184860432428a7c4962cd0379"}
Jul 14 11:38:11.490837 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:11.490811    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4f703685-4285-43da-a074-362a513a32b1-metrics-tls\") pod \"dns-default-5qmr8\" (UID: \"4f703685-4285-43da-a074-362a513a32b1\") " pod="openshift-dns/dns-default-5qmr8"
Jul 14 11:38:11.490936 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:38:11.490925    2551 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 14 11:38:11.490984 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:38:11.490980    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f703685-4285-43da-a074-362a513a32b1-metrics-tls podName:4f703685-4285-43da-a074-362a513a32b1 nodeName:}" failed. No retries permitted until 2026-07-14 11:38:19.490967025 +0000 UTC m=+50.722776299 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4f703685-4285-43da-a074-362a513a32b1-metrics-tls") pod "dns-default-5qmr8" (UID: "4f703685-4285-43da-a074-362a513a32b1") : secret "dns-default-metrics-tls" not found
Jul 14 11:38:11.503598 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:11.503392    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bb5d66b49-vmx85" podStartSLOduration=36.374532683 podStartE2EDuration="41.503372759s" totalImagesPullingTime="5.128840076s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 11:37:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 11:38:11.502298002 +0000 UTC m=+42.734107302" watchObservedRunningTime="2026-07-14 11:38:11.503372759 +0000 UTC m=+42.735182058"
Jul 14 11:38:11.522094 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:11.522039    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-x8b5w" podStartSLOduration=3.827314515 podStartE2EDuration="8.522024043s" totalImagesPullingTime="4.694709528s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 11:38:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 11:38:11.521203082 +0000 UTC m=+42.753012375" watchObservedRunningTime="2026-07-14 11:38:11.522024043 +0000 UTC m=+42.753833353"
Jul 14 11:38:12.567932 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:12.567892    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:38:12.661684 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:12.661650    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:38:12.762461 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:12.762407    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:38:13.573015 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:13.572967    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:38:13.689351 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:13.689313    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:38:13.798063 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:13.797981    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:38:14.489679 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:14.489643    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d574df866-thdlk" event={"ID":"3c59746c-ccd0-441f-bea7-f22762395f47","Type":"ContainerStarted","Data":"b3cefa29f77cc256c0ceb31a7c230030e52879d7db41776ef1f471a1e2f5a3b2"}
Jul 14 11:38:16.560321 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:16.560289    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:38:19.452745 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:19.452696    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/41b152d0-1171-4eb6-82eb-40e5d42a76b5-registry-tls\") pod \"image-registry-c9cf54fd5-7xknq\" (UID: \"41b152d0-1171-4eb6-82eb-40e5d42a76b5\") " pod="openshift-image-registry/image-registry-c9cf54fd5-7xknq"
Jul 14 11:38:19.453134 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:38:19.452856    2551 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 14 11:38:19.453134 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:38:19.452878    2551 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-c9cf54fd5-7xknq: secret "image-registry-tls" not found
Jul 14 11:38:19.453134 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:38:19.452947    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/41b152d0-1171-4eb6-82eb-40e5d42a76b5-registry-tls podName:41b152d0-1171-4eb6-82eb-40e5d42a76b5 nodeName:}" failed. No retries permitted until 2026-07-14 11:38:35.452930685 +0000 UTC m=+66.684739962 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/41b152d0-1171-4eb6-82eb-40e5d42a76b5-registry-tls") pod "image-registry-c9cf54fd5-7xknq" (UID: "41b152d0-1171-4eb6-82eb-40e5d42a76b5") : secret "image-registry-tls" not found
Jul 14 11:38:19.554011 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:19.553915    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4f703685-4285-43da-a074-362a513a32b1-metrics-tls\") pod \"dns-default-5qmr8\" (UID: \"4f703685-4285-43da-a074-362a513a32b1\") " pod="openshift-dns/dns-default-5qmr8"
Jul 14 11:38:19.554157 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:38:19.554033    2551 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 14 11:38:19.554157 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:38:19.554085    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f703685-4285-43da-a074-362a513a32b1-metrics-tls podName:4f703685-4285-43da-a074-362a513a32b1 nodeName:}" failed. No retries permitted until 2026-07-14 11:38:35.554071299 +0000 UTC m=+66.785880578 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4f703685-4285-43da-a074-362a513a32b1-metrics-tls") pod "dns-default-5qmr8" (UID: "4f703685-4285-43da-a074-362a513a32b1") : secret "dns-default-metrics-tls" not found
Jul 14 11:38:21.938010 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:21.937972    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:38:22.036627 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:22.036593    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:38:25.173446 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:25.173409    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:38:25.274181 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:25.274140    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:38:25.883923 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:25.883888    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:38:25.983684 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:25.983657    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:38:26.867147 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:26.867060    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:38:26.986243 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:26.986210    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:38:28.683247 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:28.683212    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:38:28.796330 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:28.796294    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:38:30.441551 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:30.441520    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cp57b"
Jul 14 11:38:30.475360 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:30.475305    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d574df866-thdlk" podStartSLOduration=52.614707105 podStartE2EDuration="1m0.475286092s" totalImagesPullingTime="7.860578987s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 11:37:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 11:38:14.510351445 +0000 UTC m=+45.742160741" watchObservedRunningTime="2026-07-14 11:38:30.475286092 +0000 UTC m=+61.707095368"
Jul 14 11:38:30.612940 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:30.612904    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:38:30.699162 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:30.699081    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:38:30.775834 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:30.775799    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:38:30.844019 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:30.843983    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:38:33.958267 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:33.958222    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5eafaa1-c985-4204-809a-3e9a52c96e55-metrics-certs\") pod \"network-metrics-daemon-66zbn\" (UID: \"a5eafaa1-c985-4204-809a-3e9a52c96e55\") " pod="openshift-multus/network-metrics-daemon-66zbn"
Jul 14 11:38:33.958756 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:33.958309    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3360b04a-23e1-49f9-84e4-11186ccde5e7-original-pull-secret\") pod \"global-pull-secret-syncer-75pwf\" (UID: \"3360b04a-23e1-49f9-84e4-11186ccde5e7\") " pod="kube-system/global-pull-secret-syncer-75pwf"
Jul 14 11:38:33.961845 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:33.961821    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 14 11:38:33.961845 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:33.961838    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 14 11:38:33.969246 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:38:33.969225    2551 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 14 11:38:33.969309 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:38:33.969293    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5eafaa1-c985-4204-809a-3e9a52c96e55-metrics-certs podName:a5eafaa1-c985-4204-809a-3e9a52c96e55 nodeName:}" failed. No retries permitted until 2026-07-14 11:39:37.969273244 +0000 UTC m=+129.201082518 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a5eafaa1-c985-4204-809a-3e9a52c96e55-metrics-certs") pod "network-metrics-daemon-66zbn" (UID: "a5eafaa1-c985-4204-809a-3e9a52c96e55") : secret "metrics-daemon-secret" not found
Jul 14 11:38:33.972874 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:33.972846    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3360b04a-23e1-49f9-84e4-11186ccde5e7-original-pull-secret\") pod \"global-pull-secret-syncer-75pwf\" (UID: \"3360b04a-23e1-49f9-84e4-11186ccde5e7\") " pod="kube-system/global-pull-secret-syncer-75pwf"
Jul 14 11:38:33.996154 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:33.996118    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-75pwf"
Jul 14 11:38:34.059076 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:34.059020    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77qmm\" (UniqueName: \"kubernetes.io/projected/2f44b67b-26da-4ab3-9dfa-343ad6334e4f-kube-api-access-77qmm\") pod \"network-check-target-4zvrn\" (UID: \"2f44b67b-26da-4ab3-9dfa-343ad6334e4f\") " pod="openshift-network-diagnostics/network-check-target-4zvrn"
Jul 14 11:38:34.062772 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:34.062746    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 14 11:38:34.072746 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:34.072718    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 14 11:38:34.082413 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:34.082386    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-77qmm\" (UniqueName: \"kubernetes.io/projected/2f44b67b-26da-4ab3-9dfa-343ad6334e4f-kube-api-access-77qmm\") pod \"network-check-target-4zvrn\" (UID: \"2f44b67b-26da-4ab3-9dfa-343ad6334e4f\") " pod="openshift-network-diagnostics/network-check-target-4zvrn"
Jul 14 11:38:34.127556 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:34.127525    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-75pwf"]
Jul 14 11:38:34.131308 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:38:34.131273    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3360b04a_23e1_49f9_84e4_11186ccde5e7.slice/crio-ddec72e7dcc9e72b2248450fb073cbb6b0de31491d06a7f804dcfe7cf0558c09 WatchSource:0}: Error finding container ddec72e7dcc9e72b2248450fb073cbb6b0de31491d06a7f804dcfe7cf0558c09: Status 404 returned error can't find the container with id ddec72e7dcc9e72b2248450fb073cbb6b0de31491d06a7f804dcfe7cf0558c09
Jul 14 11:38:34.200207 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:34.200176    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-9256t\""
Jul 14 11:38:34.207981 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:34.207953    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4zvrn"
Jul 14 11:38:34.337847 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:34.337808    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-4zvrn"]
Jul 14 11:38:34.340501 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:38:34.340476    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f44b67b_26da_4ab3_9dfa_343ad6334e4f.slice/crio-e453e5b70573b2384c8d7dacaefb22db1675cb86a4751ee4f9875c209c13a6fa WatchSource:0}: Error finding container e453e5b70573b2384c8d7dacaefb22db1675cb86a4751ee4f9875c209c13a6fa: Status 404 returned error can't find the container with id e453e5b70573b2384c8d7dacaefb22db1675cb86a4751ee4f9875c209c13a6fa
Jul 14 11:38:34.539344 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:34.539255    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-75pwf" event={"ID":"3360b04a-23e1-49f9-84e4-11186ccde5e7","Type":"ContainerStarted","Data":"ddec72e7dcc9e72b2248450fb073cbb6b0de31491d06a7f804dcfe7cf0558c09"}
Jul 14 11:38:34.540241 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:34.540217    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-4zvrn" event={"ID":"2f44b67b-26da-4ab3-9dfa-343ad6334e4f","Type":"ContainerStarted","Data":"e453e5b70573b2384c8d7dacaefb22db1675cb86a4751ee4f9875c209c13a6fa"}
Jul 14 11:38:35.469663 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:35.469618    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/41b152d0-1171-4eb6-82eb-40e5d42a76b5-registry-tls\") pod \"image-registry-c9cf54fd5-7xknq\" (UID: \"41b152d0-1171-4eb6-82eb-40e5d42a76b5\") " pod="openshift-image-registry/image-registry-c9cf54fd5-7xknq"
Jul 14 11:38:35.470140 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:38:35.469795    2551 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 14 11:38:35.470140 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:38:35.469819    2551 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-c9cf54fd5-7xknq: secret "image-registry-tls" not found
Jul 14 11:38:35.470140 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:38:35.469892    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/41b152d0-1171-4eb6-82eb-40e5d42a76b5-registry-tls podName:41b152d0-1171-4eb6-82eb-40e5d42a76b5 nodeName:}" failed. No retries permitted until 2026-07-14 11:39:07.469871134 +0000 UTC m=+98.701680432 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/41b152d0-1171-4eb6-82eb-40e5d42a76b5-registry-tls") pod "image-registry-c9cf54fd5-7xknq" (UID: "41b152d0-1171-4eb6-82eb-40e5d42a76b5") : secret "image-registry-tls" not found
Jul 14 11:38:35.570381 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:35.570340    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4f703685-4285-43da-a074-362a513a32b1-metrics-tls\") pod \"dns-default-5qmr8\" (UID: \"4f703685-4285-43da-a074-362a513a32b1\") " pod="openshift-dns/dns-default-5qmr8"
Jul 14 11:38:35.570600 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:38:35.570533    2551 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 14 11:38:35.570669 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:38:35.570625    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f703685-4285-43da-a074-362a513a32b1-metrics-tls podName:4f703685-4285-43da-a074-362a513a32b1 nodeName:}" failed. No retries permitted until 2026-07-14 11:39:07.570601341 +0000 UTC m=+98.802410618 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4f703685-4285-43da-a074-362a513a32b1-metrics-tls") pod "dns-default-5qmr8" (UID: "4f703685-4285-43da-a074-362a513a32b1") : secret "dns-default-metrics-tls" not found
Jul 14 11:38:41.178216 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:41.178185    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:38:41.277872 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:41.277823    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:38:41.564855 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:41.564809    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-75pwf" event={"ID":"3360b04a-23e1-49f9-84e4-11186ccde5e7","Type":"ContainerStarted","Data":"91a8e5d020d4b9f236af4afdf2ce9d0a454e8695a66320623552c9018b93dc9a"}
Jul 14 11:38:41.566248 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:41.566216    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-698bdff498-sjflm" event={"ID":"0a0efcd7-960f-4f3e-b9e0-b7bd5f19a743","Type":"ContainerStarted","Data":"c051a18651ba3dae3a316cd81c51fd69b73238316a47b7d7892443b6c5dff6d4"}
Jul 14 11:38:41.566370 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:41.566282    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-698bdff498-sjflm"
Jul 14 11:38:41.567638 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:41.567610    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-4zvrn" event={"ID":"2f44b67b-26da-4ab3-9dfa-343ad6334e4f","Type":"ContainerStarted","Data":"1cec9c4e6ff698bc9eef5bed7ff407eb7a639303b694b669a2272cbdd45f6e59"}
Jul 14 11:38:41.567740 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:41.567666    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-4zvrn"
Jul 14 11:38:41.568191 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:41.568176    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-698bdff498-sjflm"
Jul 14 11:38:41.583791 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:41.583745    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-75pwf" podStartSLOduration=65.640196464 podStartE2EDuration="1m12.583730458s" totalImagesPullingTime="6.943533994s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 11:37:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 11:38:41.583437465 +0000 UTC m=+72.815246761" watchObservedRunningTime="2026-07-14 11:38:41.583730458 +0000 UTC m=+72.815539753"
Jul 14 11:38:41.603692 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:41.603644    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-698bdff498-sjflm" podStartSLOduration=36.176121407 podStartE2EDuration="1m11.603626311s" totalImagesPullingTime="35.427504904s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 11:37:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 11:38:41.602771932 +0000 UTC m=+72.834581229" watchObservedRunningTime="2026-07-14 11:38:41.603626311 +0000 UTC m=+72.835435604"
Jul 14 11:38:41.623379 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:41.623326    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-4zvrn" podStartSLOduration=65.913732219 podStartE2EDuration="1m12.62330709s" totalImagesPullingTime="6.709574871s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 11:37:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 11:38:41.622426683 +0000 UTC m=+72.854235979" watchObservedRunningTime="2026-07-14 11:38:41.62330709 +0000 UTC m=+72.855116387"
Jul 14 11:38:47.875411 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:47.875371    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:38:48.032474 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:48.032440    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:38:50.739486 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:50.739365    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:38:50.879415 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:50.879374    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:38:51.799519 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:51.799479    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:38:51.901423 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:51.901388    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:38:53.922707 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:53.922662    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:38:56.162912 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:56.162873    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-dwk5x"]
Jul 14 11:38:56.167294 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:56.167269    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dwk5x"
Jul 14 11:38:56.170531 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:56.170501    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 14 11:38:56.170712 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:56.170693    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 14 11:38:56.170776 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:56.170749    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 14 11:38:56.170776 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:56.170756    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mf86b\""
Jul 14 11:38:56.172546 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:56.172526    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 14 11:38:56.183774 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:56.183738    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dwk5x"]
Jul 14 11:38:56.217696 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:56.217659    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:38:56.219607 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:56.219561    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3a2ef944-3d8e-41e3-a29f-5c3944611f58-crio-socket\") pod \"insights-runtime-extractor-dwk5x\" (UID: \"3a2ef944-3d8e-41e3-a29f-5c3944611f58\") " pod="openshift-insights/insights-runtime-extractor-dwk5x"
Jul 14 11:38:56.219744 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:56.219630    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3a2ef944-3d8e-41e3-a29f-5c3944611f58-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dwk5x\" (UID: \"3a2ef944-3d8e-41e3-a29f-5c3944611f58\") " pod="openshift-insights/insights-runtime-extractor-dwk5x"
Jul 14 11:38:56.219744 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:56.219676    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3a2ef944-3d8e-41e3-a29f-5c3944611f58-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dwk5x\" (UID: \"3a2ef944-3d8e-41e3-a29f-5c3944611f58\") " pod="openshift-insights/insights-runtime-extractor-dwk5x"
Jul 14 11:38:56.219744 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:56.219706    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3a2ef944-3d8e-41e3-a29f-5c3944611f58-data-volume\") pod \"insights-runtime-extractor-dwk5x\" (UID: \"3a2ef944-3d8e-41e3-a29f-5c3944611f58\") " pod="openshift-insights/insights-runtime-extractor-dwk5x"
Jul 14 11:38:56.219744 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:56.219723    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7p4hb\" (UniqueName: \"kubernetes.io/projected/3a2ef944-3d8e-41e3-a29f-5c3944611f58-kube-api-access-7p4hb\") pod \"insights-runtime-extractor-dwk5x\" (UID: \"3a2ef944-3d8e-41e3-a29f-5c3944611f58\") " pod="openshift-insights/insights-runtime-extractor-dwk5x"
Jul 14 11:38:56.297833 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:56.297798    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:38:56.320321 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:56.320285    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3a2ef944-3d8e-41e3-a29f-5c3944611f58-crio-socket\") pod \"insights-runtime-extractor-dwk5x\" (UID: \"3a2ef944-3d8e-41e3-a29f-5c3944611f58\") " pod="openshift-insights/insights-runtime-extractor-dwk5x"
Jul 14 11:38:56.320466 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:56.320333    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3a2ef944-3d8e-41e3-a29f-5c3944611f58-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dwk5x\" (UID: \"3a2ef944-3d8e-41e3-a29f-5c3944611f58\") " pod="openshift-insights/insights-runtime-extractor-dwk5x"
Jul 14 11:38:56.320466 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:38:56.320433    2551 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 14 11:38:56.320466 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:56.320432    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3a2ef944-3d8e-41e3-a29f-5c3944611f58-crio-socket\") pod \"insights-runtime-extractor-dwk5x\" (UID: \"3a2ef944-3d8e-41e3-a29f-5c3944611f58\") " pod="openshift-insights/insights-runtime-extractor-dwk5x"
Jul 14 11:38:56.320639 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:38:56.320490    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a2ef944-3d8e-41e3-a29f-5c3944611f58-insights-runtime-extractor-tls podName:3a2ef944-3d8e-41e3-a29f-5c3944611f58 nodeName:}" failed. No retries permitted until 2026-07-14 11:38:56.820476733 +0000 UTC m=+88.052286007 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3a2ef944-3d8e-41e3-a29f-5c3944611f58-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dwk5x" (UID: "3a2ef944-3d8e-41e3-a29f-5c3944611f58") : secret "insights-runtime-extractor-tls" not found
Jul 14 11:38:56.320639 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:56.320508    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3a2ef944-3d8e-41e3-a29f-5c3944611f58-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dwk5x\" (UID: \"3a2ef944-3d8e-41e3-a29f-5c3944611f58\") " pod="openshift-insights/insights-runtime-extractor-dwk5x"
Jul 14 11:38:56.320639 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:56.320541    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3a2ef944-3d8e-41e3-a29f-5c3944611f58-data-volume\") pod \"insights-runtime-extractor-dwk5x\" (UID: \"3a2ef944-3d8e-41e3-a29f-5c3944611f58\") " pod="openshift-insights/insights-runtime-extractor-dwk5x"
Jul 14 11:38:56.320639 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:56.320557    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p4hb\" (UniqueName: \"kubernetes.io/projected/3a2ef944-3d8e-41e3-a29f-5c3944611f58-kube-api-access-7p4hb\") pod \"insights-runtime-extractor-dwk5x\" (UID: \"3a2ef944-3d8e-41e3-a29f-5c3944611f58\") " pod="openshift-insights/insights-runtime-extractor-dwk5x"
Jul 14 11:38:56.320639 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:38:56.320623    2551 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 14 11:38:56.320824 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:38:56.320688    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3a2ef944-3d8e-41e3-a29f-5c3944611f58-kube-rbac-proxy-cm podName:3a2ef944-3d8e-41e3-a29f-5c3944611f58 nodeName:}" failed. No retries permitted until 2026-07-14 11:38:56.820669739 +0000 UTC m=+88.052479035 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3a2ef944-3d8e-41e3-a29f-5c3944611f58-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dwk5x" (UID: "3a2ef944-3d8e-41e3-a29f-5c3944611f58") : configmap "kube-rbac-proxy" not found
Jul 14 11:38:56.320862 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:56.320843    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3a2ef944-3d8e-41e3-a29f-5c3944611f58-data-volume\") pod \"insights-runtime-extractor-dwk5x\" (UID: \"3a2ef944-3d8e-41e3-a29f-5c3944611f58\") " pod="openshift-insights/insights-runtime-extractor-dwk5x"
Jul 14 11:38:56.331643 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:56.331610    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p4hb\" (UniqueName: \"kubernetes.io/projected/3a2ef944-3d8e-41e3-a29f-5c3944611f58-kube-api-access-7p4hb\") pod \"insights-runtime-extractor-dwk5x\" (UID: \"3a2ef944-3d8e-41e3-a29f-5c3944611f58\") " pod="openshift-insights/insights-runtime-extractor-dwk5x"
Jul 14 11:38:56.825598 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:56.825551    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3a2ef944-3d8e-41e3-a29f-5c3944611f58-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dwk5x\" (UID: \"3a2ef944-3d8e-41e3-a29f-5c3944611f58\") " pod="openshift-insights/insights-runtime-extractor-dwk5x"
Jul 14 11:38:56.825784 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:38:56.825705    2551 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 14 11:38:56.825784 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:56.825741    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3a2ef944-3d8e-41e3-a29f-5c3944611f58-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dwk5x\" (UID: \"3a2ef944-3d8e-41e3-a29f-5c3944611f58\") " pod="openshift-insights/insights-runtime-extractor-dwk5x"
Jul 14 11:38:56.825784 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:38:56.825762    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a2ef944-3d8e-41e3-a29f-5c3944611f58-insights-runtime-extractor-tls podName:3a2ef944-3d8e-41e3-a29f-5c3944611f58 nodeName:}" failed. No retries permitted until 2026-07-14 11:38:57.825747039 +0000 UTC m=+89.057556313 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3a2ef944-3d8e-41e3-a29f-5c3944611f58-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dwk5x" (UID: "3a2ef944-3d8e-41e3-a29f-5c3944611f58") : secret "insights-runtime-extractor-tls" not found
Jul 14 11:38:56.825892 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:38:56.825797    2551 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 14 11:38:56.825892 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:38:56.825852    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3a2ef944-3d8e-41e3-a29f-5c3944611f58-kube-rbac-proxy-cm podName:3a2ef944-3d8e-41e3-a29f-5c3944611f58 nodeName:}" failed. No retries permitted until 2026-07-14 11:38:57.825840125 +0000 UTC m=+89.057649399 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3a2ef944-3d8e-41e3-a29f-5c3944611f58-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dwk5x" (UID: "3a2ef944-3d8e-41e3-a29f-5c3944611f58") : configmap "kube-rbac-proxy" not found
Jul 14 11:38:57.668650 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:57.668612    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:38:57.766305 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:57.766256    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:38:57.834436 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:57.834397    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3a2ef944-3d8e-41e3-a29f-5c3944611f58-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dwk5x\" (UID: \"3a2ef944-3d8e-41e3-a29f-5c3944611f58\") " pod="openshift-insights/insights-runtime-extractor-dwk5x"
Jul 14 11:38:57.834532 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:57.834463    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3a2ef944-3d8e-41e3-a29f-5c3944611f58-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dwk5x\" (UID: \"3a2ef944-3d8e-41e3-a29f-5c3944611f58\") " pod="openshift-insights/insights-runtime-extractor-dwk5x"
Jul 14 11:38:57.834600 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:38:57.834552    2551 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 14 11:38:57.834600 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:38:57.834552    2551 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 14 11:38:57.834688 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:38:57.834633    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3a2ef944-3d8e-41e3-a29f-5c3944611f58-kube-rbac-proxy-cm podName:3a2ef944-3d8e-41e3-a29f-5c3944611f58 nodeName:}" failed. No retries permitted until 2026-07-14 11:38:59.834615724 +0000 UTC m=+91.066425005 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3a2ef944-3d8e-41e3-a29f-5c3944611f58-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dwk5x" (UID: "3a2ef944-3d8e-41e3-a29f-5c3944611f58") : configmap "kube-rbac-proxy" not found
Jul 14 11:38:57.834688 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:38:57.834648    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a2ef944-3d8e-41e3-a29f-5c3944611f58-insights-runtime-extractor-tls podName:3a2ef944-3d8e-41e3-a29f-5c3944611f58 nodeName:}" failed. No retries permitted until 2026-07-14 11:38:59.834642189 +0000 UTC m=+91.066451463 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3a2ef944-3d8e-41e3-a29f-5c3944611f58-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dwk5x" (UID: "3a2ef944-3d8e-41e3-a29f-5c3944611f58") : secret "insights-runtime-extractor-tls" not found
Jul 14 11:38:57.847070 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:57.847019    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:38:58.248820 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:58.248793    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mldq8_e356b532-03eb-4760-93eb-a731dbc0f142/dns-node-resolver/0.log"
Jul 14 11:38:58.847815 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:58.847786    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-27fdx_8590baa3-db08-49ef-8d26-3f551d6954e4/node-ca/0.log"
Jul 14 11:38:59.239836 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:59.239800    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:38:59.389309 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:59.389274    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:38:59.642647 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:59.642543    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:38:59.781852 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:59.781820    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:38:59.850735 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:59.850694    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3a2ef944-3d8e-41e3-a29f-5c3944611f58-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dwk5x\" (UID: \"3a2ef944-3d8e-41e3-a29f-5c3944611f58\") " pod="openshift-insights/insights-runtime-extractor-dwk5x"
Jul 14 11:38:59.851133 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:59.850762    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3a2ef944-3d8e-41e3-a29f-5c3944611f58-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dwk5x\" (UID: \"3a2ef944-3d8e-41e3-a29f-5c3944611f58\") " pod="openshift-insights/insights-runtime-extractor-dwk5x"
Jul 14 11:38:59.851133 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:38:59.850843    2551 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 14 11:38:59.851133 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:38:59.850915    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a2ef944-3d8e-41e3-a29f-5c3944611f58-insights-runtime-extractor-tls podName:3a2ef944-3d8e-41e3-a29f-5c3944611f58 nodeName:}" failed. No retries permitted until 2026-07-14 11:39:03.850897765 +0000 UTC m=+95.082707038 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3a2ef944-3d8e-41e3-a29f-5c3944611f58-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dwk5x" (UID: "3a2ef944-3d8e-41e3-a29f-5c3944611f58") : secret "insights-runtime-extractor-tls" not found
Jul 14 11:38:59.851133 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:38:59.850852    2551 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 14 11:38:59.851133 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:38:59.850970    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3a2ef944-3d8e-41e3-a29f-5c3944611f58-kube-rbac-proxy-cm podName:3a2ef944-3d8e-41e3-a29f-5c3944611f58 nodeName:}" failed. No retries permitted until 2026-07-14 11:39:03.850955634 +0000 UTC m=+95.082764915 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3a2ef944-3d8e-41e3-a29f-5c3944611f58-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dwk5x" (UID: "3a2ef944-3d8e-41e3-a29f-5c3944611f58") : configmap "kube-rbac-proxy" not found
Jul 14 11:38:59.931861 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:38:59.931769    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:39:00.054471 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:00.054439    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-x8b5w_2585982a-d4d0-4d80-9b85-fbfe27886ab2/serve-healthcheck-canary/0.log"
Jul 14 11:39:01.572021 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:01.571981    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:39:01.678972 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:01.678943    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:39:03.154023 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:03.153989    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:39:03.884496 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:03.884451    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3a2ef944-3d8e-41e3-a29f-5c3944611f58-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dwk5x\" (UID: \"3a2ef944-3d8e-41e3-a29f-5c3944611f58\") " pod="openshift-insights/insights-runtime-extractor-dwk5x"
Jul 14 11:39:03.884695 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:03.884554    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3a2ef944-3d8e-41e3-a29f-5c3944611f58-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dwk5x\" (UID: \"3a2ef944-3d8e-41e3-a29f-5c3944611f58\") " pod="openshift-insights/insights-runtime-extractor-dwk5x"
Jul 14 11:39:03.884695 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:39:03.884601    2551 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 14 11:39:03.884695 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:39:03.884674    2551 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 14 11:39:03.884811 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:39:03.884675    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3a2ef944-3d8e-41e3-a29f-5c3944611f58-kube-rbac-proxy-cm podName:3a2ef944-3d8e-41e3-a29f-5c3944611f58 nodeName:}" failed. No retries permitted until 2026-07-14 11:39:11.884655263 +0000 UTC m=+103.116464549 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3a2ef944-3d8e-41e3-a29f-5c3944611f58-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dwk5x" (UID: "3a2ef944-3d8e-41e3-a29f-5c3944611f58") : configmap "kube-rbac-proxy" not found
Jul 14 11:39:03.884811 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:39:03.884721    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a2ef944-3d8e-41e3-a29f-5c3944611f58-insights-runtime-extractor-tls podName:3a2ef944-3d8e-41e3-a29f-5c3944611f58 nodeName:}" failed. No retries permitted until 2026-07-14 11:39:11.88470991 +0000 UTC m=+103.116519185 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3a2ef944-3d8e-41e3-a29f-5c3944611f58-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dwk5x" (UID: "3a2ef944-3d8e-41e3-a29f-5c3944611f58") : secret "insights-runtime-extractor-tls" not found
Jul 14 11:39:05.535221 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:05.535185    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:39:05.761180 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:05.761141    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:39:06.098124 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:06.098084    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:39:06.722644 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:06.722611    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:39:07.420974 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:07.420939    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:39:07.515783 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:07.515754    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/41b152d0-1171-4eb6-82eb-40e5d42a76b5-registry-tls\") pod \"image-registry-c9cf54fd5-7xknq\" (UID: \"41b152d0-1171-4eb6-82eb-40e5d42a76b5\") " pod="openshift-image-registry/image-registry-c9cf54fd5-7xknq"
Jul 14 11:39:07.515929 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:39:07.515908    2551 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 14 11:39:07.515966 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:39:07.515935    2551 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-c9cf54fd5-7xknq: secret "image-registry-tls" not found
Jul 14 11:39:07.516007 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:39:07.515997    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/41b152d0-1171-4eb6-82eb-40e5d42a76b5-registry-tls podName:41b152d0-1171-4eb6-82eb-40e5d42a76b5 nodeName:}" failed. No retries permitted until 2026-07-14 11:40:11.515980677 +0000 UTC m=+162.747789965 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/41b152d0-1171-4eb6-82eb-40e5d42a76b5-registry-tls") pod "image-registry-c9cf54fd5-7xknq" (UID: "41b152d0-1171-4eb6-82eb-40e5d42a76b5") : secret "image-registry-tls" not found
Jul 14 11:39:07.561021 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:07.560985    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:39:07.617278 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:07.617233    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4f703685-4285-43da-a074-362a513a32b1-metrics-tls\") pod \"dns-default-5qmr8\" (UID: \"4f703685-4285-43da-a074-362a513a32b1\") " pod="openshift-dns/dns-default-5qmr8"
Jul 14 11:39:07.617462 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:39:07.617376    2551 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 14 11:39:07.617462 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:39:07.617441    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f703685-4285-43da-a074-362a513a32b1-metrics-tls podName:4f703685-4285-43da-a074-362a513a32b1 nodeName:}" failed. No retries permitted until 2026-07-14 11:40:11.61742516 +0000 UTC m=+162.849234434 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4f703685-4285-43da-a074-362a513a32b1-metrics-tls") pod "dns-default-5qmr8" (UID: "4f703685-4285-43da-a074-362a513a32b1") : secret "dns-default-metrics-tls" not found
Jul 14 11:39:07.679705 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:07.679611    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:39:07.838554 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:07.838520    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:39:08.477365 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:08.477335    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:39:08.924509 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:08.924426    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:39:09.106124 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:09.106089    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:39:09.494941 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:09.494905    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:39:09.777614 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:09.777504    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:39:10.295920 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:10.295877    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:39:10.482600 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:10.482545    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:39:10.643619 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:10.643522    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:39:10.956354 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:10.956324    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:39:11.297179 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:11.297084    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:39:11.827439 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:11.827390    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:39:11.951285 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:11.951245    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3a2ef944-3d8e-41e3-a29f-5c3944611f58-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dwk5x\" (UID: \"3a2ef944-3d8e-41e3-a29f-5c3944611f58\") " pod="openshift-insights/insights-runtime-extractor-dwk5x"
Jul 14 11:39:11.951484 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:11.951314    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3a2ef944-3d8e-41e3-a29f-5c3944611f58-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dwk5x\" (UID: \"3a2ef944-3d8e-41e3-a29f-5c3944611f58\") " pod="openshift-insights/insights-runtime-extractor-dwk5x"
Jul 14 11:39:11.951484 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:39:11.951394    2551 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 14 11:39:11.951484 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:39:11.951477    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a2ef944-3d8e-41e3-a29f-5c3944611f58-insights-runtime-extractor-tls podName:3a2ef944-3d8e-41e3-a29f-5c3944611f58 nodeName:}" failed. No retries permitted until 2026-07-14 11:39:27.951458171 +0000 UTC m=+119.183267446 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3a2ef944-3d8e-41e3-a29f-5c3944611f58-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dwk5x" (UID: "3a2ef944-3d8e-41e3-a29f-5c3944611f58") : secret "insights-runtime-extractor-tls" not found
Jul 14 11:39:11.951890 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:11.951869    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3a2ef944-3d8e-41e3-a29f-5c3944611f58-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dwk5x\" (UID: \"3a2ef944-3d8e-41e3-a29f-5c3944611f58\") " pod="openshift-insights/insights-runtime-extractor-dwk5x"
Jul 14 11:39:12.183387 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:12.183296    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:39:12.453175 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:12.453138    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:39:12.573230 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:12.573204    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-4zvrn"
Jul 14 11:39:12.659038 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:12.659003    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:39:13.035798 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:13.035754    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:39:13.250131 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:13.250100    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:39:13.635162 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:13.635126    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:39:13.816527 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:13.816487    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:39:13.900793 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:13.900698    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:39:14.455558 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:14.455521    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:39:14.701302 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:14.701268    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:39:15.027863 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:15.027826    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:39:15.151949 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:15.151912    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:39:15.302845 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:15.302757    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:39:15.868623 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:15.868574    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:39:16.068001 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:16.067969    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:39:16.231145 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:16.231109    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:39:18.093258 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:18.093221    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:39:18.195593 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:18.195549    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:39:18.318251 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:18.318220    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:39:18.422415 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:18.422334    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:39:18.756549 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:18.756514    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:39:18.864150 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:18.864111    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:39:19.171161 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:19.171076    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:39:19.745146 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:19.745114    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:39:19.947920 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:19.947880    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:39:20.049159 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:20.049077    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:39:20.227682 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:20.227652    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:39:22.563599 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:22.563544    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:39:22.637450 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:22.637419    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:39:24.769060 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:24.769025    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:39:25.792521 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:25.792485    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:39:26.031627 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:26.031565    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:39:27.028538 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:27.028498    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:39:27.978902 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:27.978838    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3a2ef944-3d8e-41e3-a29f-5c3944611f58-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dwk5x\" (UID: \"3a2ef944-3d8e-41e3-a29f-5c3944611f58\") " pod="openshift-insights/insights-runtime-extractor-dwk5x"
Jul 14 11:39:27.981132 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:27.981112    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3a2ef944-3d8e-41e3-a29f-5c3944611f58-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dwk5x\" (UID: \"3a2ef944-3d8e-41e3-a29f-5c3944611f58\") " pod="openshift-insights/insights-runtime-extractor-dwk5x"
Jul 14 11:39:28.276844 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:28.276749    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dwk5x"
Jul 14 11:39:28.423438 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:28.423401    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dwk5x"]
Jul 14 11:39:28.427415 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:39:28.427388    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a2ef944_3d8e_41e3_a29f_5c3944611f58.slice/crio-ee3ab44d1fcfa836134d1ab939d5c05e32147e16abccb19bc55bed898f808a48 WatchSource:0}: Error finding container ee3ab44d1fcfa836134d1ab939d5c05e32147e16abccb19bc55bed898f808a48: Status 404 returned error can't find the container with id ee3ab44d1fcfa836134d1ab939d5c05e32147e16abccb19bc55bed898f808a48
Jul 14 11:39:28.710546 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:28.710514    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dwk5x" event={"ID":"3a2ef944-3d8e-41e3-a29f-5c3944611f58","Type":"ContainerStarted","Data":"ee3ab44d1fcfa836134d1ab939d5c05e32147e16abccb19bc55bed898f808a48"}
Jul 14 11:39:29.064060 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:29.063954    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:39:29.214710 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:29.214657    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:39:29.606455 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:29.606373    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:39:29.678251 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:29.678197    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:39:31.187842 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:31.187801    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:39:31.268961 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:31.268928    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:39:31.797246 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:31.797195    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:39:31.889783 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:31.889743    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:39:32.726899 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:32.726865    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dwk5x" event={"ID":"3a2ef944-3d8e-41e3-a29f-5c3944611f58","Type":"ContainerStarted","Data":"2d765dad0a8a6273a41b55e6e9673ba9726ad7fd8c84afd9ba72ff55b89a8728"}
Jul 14 11:39:33.420625 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:39:33.420525    2551 log.go:32] "PullImage from image service failed" err="rpc error: code = Unknown desc = unable to pull image or OCI artifact: pull image err: copying system image from manifest list: parsing image configuration: fetching blob: received unexpected HTTP status: 503 Service Unavailable; artifact err: provided artifact is a container image" image="quay.io/openshift/origin-insights-runtime-extractor:latest"
Jul 14 11:39:33.420821 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:39:33.420693    2551 kuberuntime_manager.go:1459] "Unhandled Error" err="container extractor start failed in pod insights-runtime-extractor-dwk5x_openshift-insights(3a2ef944-3d8e-41e3-a29f-5c3944611f58): ErrImagePull: unable to pull image or OCI artifact: pull image err: copying system image from manifest list: parsing image configuration: fetching blob: received unexpected HTTP status: 503 Service Unavailable; artifact err: provided artifact is a container image" logger="UnhandledError"
Jul 14 11:39:33.420821 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:39:33.420742    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"extractor\" with ErrImagePull: \"unable to pull image or OCI artifact: pull image err: copying system image from manifest list: parsing image configuration: fetching blob: received unexpected HTTP status: 503 Service Unavailable; artifact err: provided artifact is a container image\"" pod="openshift-insights/insights-runtime-extractor-dwk5x" podUID="3a2ef944-3d8e-41e3-a29f-5c3944611f58"
Jul 14 11:39:33.731626 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:33.731563    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dwk5x" event={"ID":"3a2ef944-3d8e-41e3-a29f-5c3944611f58","Type":"ContainerStarted","Data":"83e28dd5e547e03816e7d1900732d3e21aca1fa48043d76a01bf41621b1d3aa0"}
Jul 14 11:39:33.732261 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:39:33.732236    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"extractor\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift/origin-insights-runtime-extractor:latest\\\": ErrImagePull: unable to pull image or OCI artifact: pull image err: copying system image from manifest list: parsing image configuration: fetching blob: received unexpected HTTP status: 503 Service Unavailable; artifact err: provided artifact is a container image\"" pod="openshift-insights/insights-runtime-extractor-dwk5x" podUID="3a2ef944-3d8e-41e3-a29f-5c3944611f58"
Jul 14 11:39:34.126599 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:34.126484    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:39:34.199009 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:34.198978    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:39:34.735591 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:39:34.735540    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"extractor\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift/origin-insights-runtime-extractor:latest\\\": ErrImagePull: unable to pull image or OCI artifact: pull image err: copying system image from manifest list: parsing image configuration: fetching blob: received unexpected HTTP status: 503 Service Unavailable; artifact err: provided artifact is a container image\"" pod="openshift-insights/insights-runtime-extractor-dwk5x" podUID="3a2ef944-3d8e-41e3-a29f-5c3944611f58"
Jul 14 11:39:36.002180 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:36.002147    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:39:36.084105 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:36.084061    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:39:36.366125 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:36.366036    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:39:38.059324 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:38.059282    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5eafaa1-c985-4204-809a-3e9a52c96e55-metrics-certs\") pod \"network-metrics-daemon-66zbn\" (UID: \"a5eafaa1-c985-4204-809a-3e9a52c96e55\") " pod="openshift-multus/network-metrics-daemon-66zbn"
Jul 14 11:39:38.061651 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:38.061626    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5eafaa1-c985-4204-809a-3e9a52c96e55-metrics-certs\") pod \"network-metrics-daemon-66zbn\" (UID: \"a5eafaa1-c985-4204-809a-3e9a52c96e55\") " pod="openshift-multus/network-metrics-daemon-66zbn"
Jul 14 11:39:38.106297 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:38.106270    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-26j7k\""
Jul 14 11:39:38.114639 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:38.114605    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-66zbn"
Jul 14 11:39:38.249639 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:38.249603    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-66zbn"]
Jul 14 11:39:38.252745 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:39:38.252711    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5eafaa1_c985_4204_809a_3e9a52c96e55.slice/crio-35a728a550fd370f63768d1cf20a6403c72b0638a5af7a6f7efbf851c66cfbdf WatchSource:0}: Error finding container 35a728a550fd370f63768d1cf20a6403c72b0638a5af7a6f7efbf851c66cfbdf: Status 404 returned error can't find the container with id 35a728a550fd370f63768d1cf20a6403c72b0638a5af7a6f7efbf851c66cfbdf
Jul 14 11:39:38.747648 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:38.747615    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-66zbn" event={"ID":"a5eafaa1-c985-4204-809a-3e9a52c96e55","Type":"ContainerStarted","Data":"35a728a550fd370f63768d1cf20a6403c72b0638a5af7a6f7efbf851c66cfbdf"}
Jul 14 11:39:39.619873 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:39.619783    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:39:39.752275 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:39.752240    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-66zbn" event={"ID":"a5eafaa1-c985-4204-809a-3e9a52c96e55","Type":"ContainerStarted","Data":"adcc7ac685464a589e67914f23cf8e2ad212ae9a26621f86aa1beae9b34c0baa"}
Jul 14 11:39:39.752447 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:39.752295    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-66zbn" event={"ID":"a5eafaa1-c985-4204-809a-3e9a52c96e55","Type":"ContainerStarted","Data":"f909c210f74957843d6f2c91001d57b46c5b37eaef2f5c7e19786f961096582c"}
Jul 14 11:39:39.781344 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:39.781293    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-66zbn" podStartSLOduration=129.794152317 podStartE2EDuration="2m10.781272577s" totalImagesPullingTime="987.12026ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 11:37:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 11:39:39.78086376 +0000 UTC m=+131.012673056" watchObservedRunningTime="2026-07-14 11:39:39.781272577 +0000 UTC m=+131.013081873"
Jul 14 11:39:39.901089 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:39.901007    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:39:40.035110 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:40.035073    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:39:40.117837 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:40.117803    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:39:41.142763 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:41.142723    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:39:41.442363 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:41.442329    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:39:41.509128 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:41.509092    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-lht2c"]
Jul 14 11:39:41.512432 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:41.512412    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-lht2c"
Jul 14 11:39:41.515245 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:41.515217    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 14 11:39:41.515384 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:41.515265    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 14 11:39:41.515384 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:41.515272    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 14 11:39:41.515384 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:41.515317    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 14 11:39:41.515514 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:41.515392    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 14 11:39:41.516258 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:41.516238    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 14 11:39:41.516398 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:41.516380    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-h29qj\""
Jul 14 11:39:41.530470 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:41.530438    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:39:41.589243 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:41.589205    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8aa81994-2723-49fc-af2f-a832f71b0973-node-exporter-tls\") pod \"node-exporter-lht2c\" (UID: \"8aa81994-2723-49fc-af2f-a832f71b0973\") " pod="openshift-monitoring/node-exporter-lht2c"
Jul 14 11:39:41.589243 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:41.589249    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8aa81994-2723-49fc-af2f-a832f71b0973-node-exporter-accelerators-collector-config\") pod \"node-exporter-lht2c\" (UID: \"8aa81994-2723-49fc-af2f-a832f71b0973\") " pod="openshift-monitoring/node-exporter-lht2c"
Jul 14 11:39:41.589456 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:41.589269    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8aa81994-2723-49fc-af2f-a832f71b0973-root\") pod \"node-exporter-lht2c\" (UID: \"8aa81994-2723-49fc-af2f-a832f71b0973\") " pod="openshift-monitoring/node-exporter-lht2c"
Jul 14 11:39:41.589456 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:41.589340    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8aa81994-2723-49fc-af2f-a832f71b0973-metrics-client-ca\") pod \"node-exporter-lht2c\" (UID: \"8aa81994-2723-49fc-af2f-a832f71b0973\") " pod="openshift-monitoring/node-exporter-lht2c"
Jul 14 11:39:41.589456 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:41.589390    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsfkp\" (UniqueName: \"kubernetes.io/projected/8aa81994-2723-49fc-af2f-a832f71b0973-kube-api-access-gsfkp\") pod \"node-exporter-lht2c\" (UID: \"8aa81994-2723-49fc-af2f-a832f71b0973\") " pod="openshift-monitoring/node-exporter-lht2c"
Jul 14 11:39:41.589456 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:41.589427    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8aa81994-2723-49fc-af2f-a832f71b0973-node-exporter-wtmp\") pod \"node-exporter-lht2c\" (UID: \"8aa81994-2723-49fc-af2f-a832f71b0973\") " pod="openshift-monitoring/node-exporter-lht2c"
Jul 14 11:39:41.589456 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:41.589449    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8aa81994-2723-49fc-af2f-a832f71b0973-sys\") pod \"node-exporter-lht2c\" (UID: \"8aa81994-2723-49fc-af2f-a832f71b0973\") " pod="openshift-monitoring/node-exporter-lht2c"
Jul 14 11:39:41.589681 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:41.589471    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8aa81994-2723-49fc-af2f-a832f71b0973-node-exporter-textfile\") pod \"node-exporter-lht2c\" (UID: \"8aa81994-2723-49fc-af2f-a832f71b0973\") " pod="openshift-monitoring/node-exporter-lht2c"
Jul 14 11:39:41.589681 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:41.589487    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8aa81994-2723-49fc-af2f-a832f71b0973-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lht2c\" (UID: \"8aa81994-2723-49fc-af2f-a832f71b0973\") " pod="openshift-monitoring/node-exporter-lht2c"
Jul 14 11:39:41.634646 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:41.634567    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:39:41.690348 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:41.690313    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8aa81994-2723-49fc-af2f-a832f71b0973-node-exporter-tls\") pod \"node-exporter-lht2c\" (UID: \"8aa81994-2723-49fc-af2f-a832f71b0973\") " pod="openshift-monitoring/node-exporter-lht2c"
Jul 14 11:39:41.690540 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:41.690357    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8aa81994-2723-49fc-af2f-a832f71b0973-node-exporter-accelerators-collector-config\") pod \"node-exporter-lht2c\" (UID: \"8aa81994-2723-49fc-af2f-a832f71b0973\") " pod="openshift-monitoring/node-exporter-lht2c"
Jul 14 11:39:41.690540 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:41.690376    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8aa81994-2723-49fc-af2f-a832f71b0973-root\") pod \"node-exporter-lht2c\" (UID: \"8aa81994-2723-49fc-af2f-a832f71b0973\") " pod="openshift-monitoring/node-exporter-lht2c"
Jul 14 11:39:41.690540 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:41.690415    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8aa81994-2723-49fc-af2f-a832f71b0973-metrics-client-ca\") pod \"node-exporter-lht2c\" (UID: \"8aa81994-2723-49fc-af2f-a832f71b0973\") " pod="openshift-monitoring/node-exporter-lht2c"
Jul 14 11:39:41.690540 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:41.690444    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsfkp\" (UniqueName: \"kubernetes.io/projected/8aa81994-2723-49fc-af2f-a832f71b0973-kube-api-access-gsfkp\") pod \"node-exporter-lht2c\" (UID: \"8aa81994-2723-49fc-af2f-a832f71b0973\") " pod="openshift-monitoring/node-exporter-lht2c"
Jul 14 11:39:41.690540 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:41.690484    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8aa81994-2723-49fc-af2f-a832f71b0973-node-exporter-wtmp\") pod \"node-exporter-lht2c\" (UID: \"8aa81994-2723-49fc-af2f-a832f71b0973\") " pod="openshift-monitoring/node-exporter-lht2c"
Jul 14 11:39:41.690540 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:41.690511    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8aa81994-2723-49fc-af2f-a832f71b0973-sys\") pod \"node-exporter-lht2c\" (UID: \"8aa81994-2723-49fc-af2f-a832f71b0973\") " pod="openshift-monitoring/node-exporter-lht2c"
Jul 14 11:39:41.690890 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:41.690538    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8aa81994-2723-49fc-af2f-a832f71b0973-root\") pod \"node-exporter-lht2c\" (UID: \"8aa81994-2723-49fc-af2f-a832f71b0973\") " pod="openshift-monitoring/node-exporter-lht2c"
Jul 14 11:39:41.690890 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:41.690564    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8aa81994-2723-49fc-af2f-a832f71b0973-sys\") pod \"node-exporter-lht2c\" (UID: \"8aa81994-2723-49fc-af2f-a832f71b0973\") " pod="openshift-monitoring/node-exporter-lht2c"
Jul 14 11:39:41.690890 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:41.690605    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8aa81994-2723-49fc-af2f-a832f71b0973-node-exporter-textfile\") pod \"node-exporter-lht2c\" (UID: \"8aa81994-2723-49fc-af2f-a832f71b0973\") " pod="openshift-monitoring/node-exporter-lht2c"
Jul 14 11:39:41.690890 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:41.690639    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8aa81994-2723-49fc-af2f-a832f71b0973-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lht2c\" (UID: \"8aa81994-2723-49fc-af2f-a832f71b0973\") " pod="openshift-monitoring/node-exporter-lht2c"
Jul 14 11:39:41.690890 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:41.690774    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8aa81994-2723-49fc-af2f-a832f71b0973-node-exporter-wtmp\") pod \"node-exporter-lht2c\" (UID: \"8aa81994-2723-49fc-af2f-a832f71b0973\") " pod="openshift-monitoring/node-exporter-lht2c"
Jul 14 11:39:41.691219 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:41.691008    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8aa81994-2723-49fc-af2f-a832f71b0973-node-exporter-textfile\") pod \"node-exporter-lht2c\" (UID: \"8aa81994-2723-49fc-af2f-a832f71b0973\") " pod="openshift-monitoring/node-exporter-lht2c"
Jul 14 11:39:41.691219 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:41.691182    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8aa81994-2723-49fc-af2f-a832f71b0973-node-exporter-accelerators-collector-config\") pod \"node-exporter-lht2c\" (UID: \"8aa81994-2723-49fc-af2f-a832f71b0973\") " pod="openshift-monitoring/node-exporter-lht2c"
Jul 14 11:39:41.691344 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:41.691318    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8aa81994-2723-49fc-af2f-a832f71b0973-metrics-client-ca\") pod \"node-exporter-lht2c\" (UID: \"8aa81994-2723-49fc-af2f-a832f71b0973\") " pod="openshift-monitoring/node-exporter-lht2c"
Jul 14 11:39:41.693111 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:41.693030    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8aa81994-2723-49fc-af2f-a832f71b0973-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lht2c\" (UID: \"8aa81994-2723-49fc-af2f-a832f71b0973\") " pod="openshift-monitoring/node-exporter-lht2c"
Jul 14 11:39:41.693226 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:41.693183    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8aa81994-2723-49fc-af2f-a832f71b0973-node-exporter-tls\") pod \"node-exporter-lht2c\" (UID: \"8aa81994-2723-49fc-af2f-a832f71b0973\") " pod="openshift-monitoring/node-exporter-lht2c"
Jul 14 11:39:41.700519 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:41.700486    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsfkp\" (UniqueName: \"kubernetes.io/projected/8aa81994-2723-49fc-af2f-a832f71b0973-kube-api-access-gsfkp\") pod \"node-exporter-lht2c\" (UID: \"8aa81994-2723-49fc-af2f-a832f71b0973\") " pod="openshift-monitoring/node-exporter-lht2c"
Jul 14 11:39:41.733748 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:41.733712    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:39:41.821408 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:41.821369    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-lht2c"
Jul 14 11:39:41.831424 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:41.831386    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:39:41.833138 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:39:41.833113    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8aa81994_2723_49fc_af2f_a832f71b0973.slice/crio-6f7cb7b036ff07191882a0c7eca3a6ee29f94ca6d0329e4c89f852505950faa8 WatchSource:0}: Error finding container 6f7cb7b036ff07191882a0c7eca3a6ee29f94ca6d0329e4c89f852505950faa8: Status 404 returned error can't find the container with id 6f7cb7b036ff07191882a0c7eca3a6ee29f94ca6d0329e4c89f852505950faa8
Jul 14 11:39:41.953277 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:41.953180    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:39:42.401438 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:42.401320    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:39:42.508893 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:42.508846    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:39:42.762751 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:42.762717    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lht2c" event={"ID":"8aa81994-2723-49fc-af2f-a832f71b0973","Type":"ContainerStarted","Data":"6f7cb7b036ff07191882a0c7eca3a6ee29f94ca6d0329e4c89f852505950faa8"}
Jul 14 11:39:42.948721 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:42.948687    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:39:43.189423 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:43.189383    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:39:43.767616 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:43.767567    2551 generic.go:356] "Generic (PLEG): container finished" podID="8aa81994-2723-49fc-af2f-a832f71b0973" containerID="5cbccbe833698fb5c3fec181a61b27077b21c908377c6deb6699879526364008" exitCode=0
Jul 14 11:39:43.768006 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:43.767661    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lht2c" event={"ID":"8aa81994-2723-49fc-af2f-a832f71b0973","Type":"ContainerDied","Data":"5cbccbe833698fb5c3fec181a61b27077b21c908377c6deb6699879526364008"}
Jul 14 11:39:43.777449 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:43.777422    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:39:43.876719 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:43.876687    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:39:43.972521 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:43.972470    2551 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bb5d66b49-vmx85" podUID="6329a619-10fc-475e-88bb-b0573546fbd5" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 14 11:39:44.369451 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:44.369414    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:39:44.441873 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:44.441831    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:39:44.613043 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:44.613005    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:39:44.773952 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:44.773917    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lht2c" event={"ID":"8aa81994-2723-49fc-af2f-a832f71b0973","Type":"ContainerStarted","Data":"e1e75e6bc534df61f19935be5af4817bf711c5cbf785403d2968bac5562e2a94"}
Jul 14 11:39:44.774311 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:44.774012    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lht2c" event={"ID":"8aa81994-2723-49fc-af2f-a832f71b0973","Type":"ContainerStarted","Data":"f76b2431013a82a0df1b73295d799d53dd77c97a336018595b3811ff243e8acc"}
Jul 14 11:39:44.797895 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:44.797833    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-lht2c" podStartSLOduration=2.763819446 podStartE2EDuration="3.797817455s" totalImagesPullingTime="968.741565ms" totalInitContainerRuntime="65.256444ms" isStatefulPod=true podCreationTimestamp="2026-07-14 11:39:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 11:39:44.796237276 +0000 UTC m=+136.028046589" watchObservedRunningTime="2026-07-14 11:39:44.797817455 +0000 UTC m=+136.029626750"
Jul 14 11:39:44.875231 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:44.875197    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:39:46.184698 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:46.184664    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:39:46.253204 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:46.253168    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:39:46.363960 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:46.363923    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:39:46.465465 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:46.465367    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:39:46.565526 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:46.565482    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:39:46.975187 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:46.975154    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:39:47.086424 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:47.086381    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:39:47.855202 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:47.855153    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:39:48.220600 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:48.220555    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:39:48.345895 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:48.345855    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:39:49.796521 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:49.796483    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dwk5x" event={"ID":"3a2ef944-3d8e-41e3-a29f-5c3944611f58","Type":"ContainerStarted","Data":"cd09fbabbbf6dcd678dc162448da8d76bf345433d77486b466b6eeb8101fcc53"}
Jul 14 11:39:49.817981 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:49.817681    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-dwk5x" podStartSLOduration=32.713982845 podStartE2EDuration="53.817659133s" totalImagesPullingTime="21.103676288s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 11:38:56 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=1 observedRunningTime="2026-07-14 11:39:49.817146332 +0000 UTC m=+141.048955628" watchObservedRunningTime="2026-07-14 11:39:49.817659133 +0000 UTC m=+141.049468429"
Jul 14 11:39:52.611494 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:52.611459    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:39:52.710603 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:52.710547    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:39:53.972212 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:53.972173    2551 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bb5d66b49-vmx85" podUID="6329a619-10fc-475e-88bb-b0573546fbd5" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 14 11:39:54.048816 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:54.048778    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-c9cf54fd5-7xknq"]
Jul 14 11:39:54.049023 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:39:54.049004    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-c9cf54fd5-7xknq" podUID="41b152d0-1171-4eb6-82eb-40e5d42a76b5"
Jul 14 11:39:54.145724 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:54.145695    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:39:54.220345 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:54.220311    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:39:54.283270 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:54.283187    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:39:54.812513 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:54.812482    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-c9cf54fd5-7xknq"
Jul 14 11:39:54.818834 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:54.818810    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-c9cf54fd5-7xknq"
Jul 14 11:39:54.892492 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:54.892458    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/41b152d0-1171-4eb6-82eb-40e5d42a76b5-ca-trust-extracted\") pod \"41b152d0-1171-4eb6-82eb-40e5d42a76b5\" (UID: \"41b152d0-1171-4eb6-82eb-40e5d42a76b5\") "
Jul 14 11:39:54.892492 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:54.892497    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/41b152d0-1171-4eb6-82eb-40e5d42a76b5-trusted-ca\") pod \"41b152d0-1171-4eb6-82eb-40e5d42a76b5\" (UID: \"41b152d0-1171-4eb6-82eb-40e5d42a76b5\") "
Jul 14 11:39:54.892730 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:54.892519    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/41b152d0-1171-4eb6-82eb-40e5d42a76b5-installation-pull-secrets\") pod \"41b152d0-1171-4eb6-82eb-40e5d42a76b5\" (UID: \"41b152d0-1171-4eb6-82eb-40e5d42a76b5\") "
Jul 14 11:39:54.892730 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:54.892538    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/41b152d0-1171-4eb6-82eb-40e5d42a76b5-image-registry-private-configuration\") pod \"41b152d0-1171-4eb6-82eb-40e5d42a76b5\" (UID: \"41b152d0-1171-4eb6-82eb-40e5d42a76b5\") "
Jul 14 11:39:54.892823 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:54.892736    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/41b152d0-1171-4eb6-82eb-40e5d42a76b5-registry-certificates\") pod \"41b152d0-1171-4eb6-82eb-40e5d42a76b5\" (UID: \"41b152d0-1171-4eb6-82eb-40e5d42a76b5\") "
Jul 14 11:39:54.892823 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:54.892782    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fdwn4\" (UniqueName: \"kubernetes.io/projected/41b152d0-1171-4eb6-82eb-40e5d42a76b5-kube-api-access-fdwn4\") pod \"41b152d0-1171-4eb6-82eb-40e5d42a76b5\" (UID: \"41b152d0-1171-4eb6-82eb-40e5d42a76b5\") "
Jul 14 11:39:54.892823 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:54.892816    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/41b152d0-1171-4eb6-82eb-40e5d42a76b5-bound-sa-token\") pod \"41b152d0-1171-4eb6-82eb-40e5d42a76b5\" (UID: \"41b152d0-1171-4eb6-82eb-40e5d42a76b5\") "
Jul 14 11:39:54.892957 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:54.892834    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41b152d0-1171-4eb6-82eb-40e5d42a76b5-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "41b152d0-1171-4eb6-82eb-40e5d42a76b5" (UID: "41b152d0-1171-4eb6-82eb-40e5d42a76b5"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 11:39:54.893086 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:54.893061    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41b152d0-1171-4eb6-82eb-40e5d42a76b5-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "41b152d0-1171-4eb6-82eb-40e5d42a76b5" (UID: "41b152d0-1171-4eb6-82eb-40e5d42a76b5"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 11:39:54.893170 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:54.893067    2551 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/41b152d0-1171-4eb6-82eb-40e5d42a76b5-ca-trust-extracted\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 11:39:54.893170 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:54.893130    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41b152d0-1171-4eb6-82eb-40e5d42a76b5-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "41b152d0-1171-4eb6-82eb-40e5d42a76b5" (UID: "41b152d0-1171-4eb6-82eb-40e5d42a76b5"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 11:39:54.895048 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:54.895002    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41b152d0-1171-4eb6-82eb-40e5d42a76b5-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "41b152d0-1171-4eb6-82eb-40e5d42a76b5" (UID: "41b152d0-1171-4eb6-82eb-40e5d42a76b5"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 11:39:54.895048 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:54.895029    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41b152d0-1171-4eb6-82eb-40e5d42a76b5-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "41b152d0-1171-4eb6-82eb-40e5d42a76b5" (UID: "41b152d0-1171-4eb6-82eb-40e5d42a76b5"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 11:39:54.895216 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:54.895175    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41b152d0-1171-4eb6-82eb-40e5d42a76b5-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "41b152d0-1171-4eb6-82eb-40e5d42a76b5" (UID: "41b152d0-1171-4eb6-82eb-40e5d42a76b5"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 11:39:54.895254 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:54.895212    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41b152d0-1171-4eb6-82eb-40e5d42a76b5-kube-api-access-fdwn4" (OuterVolumeSpecName: "kube-api-access-fdwn4") pod "41b152d0-1171-4eb6-82eb-40e5d42a76b5" (UID: "41b152d0-1171-4eb6-82eb-40e5d42a76b5"). InnerVolumeSpecName "kube-api-access-fdwn4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 11:39:54.993617 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:54.993567    2551 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/41b152d0-1171-4eb6-82eb-40e5d42a76b5-bound-sa-token\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 11:39:54.993617 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:54.993610    2551 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/41b152d0-1171-4eb6-82eb-40e5d42a76b5-trusted-ca\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 11:39:54.994037 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:54.993619    2551 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/41b152d0-1171-4eb6-82eb-40e5d42a76b5-installation-pull-secrets\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 11:39:54.994037 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:54.993640    2551 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/41b152d0-1171-4eb6-82eb-40e5d42a76b5-image-registry-private-configuration\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 11:39:54.994037 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:54.993650    2551 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/41b152d0-1171-4eb6-82eb-40e5d42a76b5-registry-certificates\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 11:39:54.994037 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:54.993659    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdwn4\" (UniqueName: \"kubernetes.io/projected/41b152d0-1171-4eb6-82eb-40e5d42a76b5-kube-api-access-fdwn4\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 11:39:55.815853 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:55.815821    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-c9cf54fd5-7xknq"
Jul 14 11:39:55.854761 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:55.854731    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-c9cf54fd5-7xknq"]
Jul 14 11:39:55.859599 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:55.859551    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-c9cf54fd5-7xknq"]
Jul 14 11:39:55.900385 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:55.900346    2551 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/41b152d0-1171-4eb6-82eb-40e5d42a76b5-registry-tls\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 11:39:57.144603 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:57.144560    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:39:57.190091 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:57.190057    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41b152d0-1171-4eb6-82eb-40e5d42a76b5" path="/var/lib/kubelet/pods/41b152d0-1171-4eb6-82eb-40e5d42a76b5/volumes"
Jul 14 11:39:57.239762 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:57.239723    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:39:58.768840 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:58.768802    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:39:58.870254 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:58.870221    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:39:59.235084 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:59.235044    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:39:59.334349 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:39:59.334315    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:40:01.688348 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:40:01.688301    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:40:01.793376 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:40:01.793341    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:40:02.261246 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:40:02.261186    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:40:02.415244 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:40:02.415207    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:40:02.917291 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:40:02.917256    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:40:03.972350 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:40:03.972310    2551 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bb5d66b49-vmx85" podUID="6329a619-10fc-475e-88bb-b0573546fbd5" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 14 11:40:03.972786 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:40:03.972434    2551 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bb5d66b49-vmx85"
Jul 14 11:40:03.973074 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:40:03.973034    2551 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"b51d86b6c02671183dd10537e30cb824093ca331cddae7a769debbede03f33bb"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bb5d66b49-vmx85" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Jul 14 11:40:03.973137 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:40:03.973095    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bb5d66b49-vmx85" podUID="6329a619-10fc-475e-88bb-b0573546fbd5" containerName="service-proxy" containerID="cri-o://b51d86b6c02671183dd10537e30cb824093ca331cddae7a769debbede03f33bb" gracePeriod=30
Jul 14 11:40:04.207952 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:40:04.207915    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:40:04.303770 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:40:04.303690    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:40:04.846831 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:40:04.846797    2551 generic.go:356] "Generic (PLEG): container finished" podID="6329a619-10fc-475e-88bb-b0573546fbd5" containerID="b51d86b6c02671183dd10537e30cb824093ca331cddae7a769debbede03f33bb" exitCode=2
Jul 14 11:40:04.847025 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:40:04.846884    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bb5d66b49-vmx85" event={"ID":"6329a619-10fc-475e-88bb-b0573546fbd5","Type":"ContainerDied","Data":"b51d86b6c02671183dd10537e30cb824093ca331cddae7a769debbede03f33bb"}
Jul 14 11:40:04.847025 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:40:04.846944    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bb5d66b49-vmx85" event={"ID":"6329a619-10fc-475e-88bb-b0573546fbd5","Type":"ContainerStarted","Data":"f3b3480d7a1e99b9e7c206b087456b56e4f9b97a623637b87efeb9e6d87705be"}
Jul 14 11:40:06.708936 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:40:06.708885    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-5qmr8" podUID="4f703685-4285-43da-a074-362a513a32b1"
Jul 14 11:40:06.852814 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:40:06.852782    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5qmr8"
Jul 14 11:40:09.188068 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:40:09.188037    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:40:09.289091 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:40:09.289054    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:40:09.366918 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:40:09.366876    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:40:11.629535 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:40:11.629491    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4f703685-4285-43da-a074-362a513a32b1-metrics-tls\") pod \"dns-default-5qmr8\" (UID: \"4f703685-4285-43da-a074-362a513a32b1\") " pod="openshift-dns/dns-default-5qmr8"
Jul 14 11:40:11.631909 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:40:11.631880    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4f703685-4285-43da-a074-362a513a32b1-metrics-tls\") pod \"dns-default-5qmr8\" (UID: \"4f703685-4285-43da-a074-362a513a32b1\") " pod="openshift-dns/dns-default-5qmr8"
Jul 14 11:40:11.656754 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:40:11.656721    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-9dtrp\""
Jul 14 11:40:11.664659 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:40:11.664630    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5qmr8"
Jul 14 11:40:11.823506 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:40:11.823418    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5qmr8"]
Jul 14 11:40:11.826164 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:40:11.826140    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f703685_4285_43da_a074_362a513a32b1.slice/crio-ef07d26cb61bb544e6ccd64a58d3178bfb4eb3e1fb7495c1a173fef7525b6e28 WatchSource:0}: Error finding container ef07d26cb61bb544e6ccd64a58d3178bfb4eb3e1fb7495c1a173fef7525b6e28: Status 404 returned error can't find the container with id ef07d26cb61bb544e6ccd64a58d3178bfb4eb3e1fb7495c1a173fef7525b6e28
Jul 14 11:40:11.873278 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:40:11.873236    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5qmr8" event={"ID":"4f703685-4285-43da-a074-362a513a32b1","Type":"ContainerStarted","Data":"ef07d26cb61bb544e6ccd64a58d3178bfb4eb3e1fb7495c1a173fef7525b6e28"}
Jul 14 11:40:13.327371 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:40:13.327338    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:40:13.428288 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:40:13.428248    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:40:13.528863 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:40:13.528826    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:40:13.758093 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:40:13.758051    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:40:14.139592 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:40:14.139493    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:40:14.885474 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:40:14.885432    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5qmr8" event={"ID":"4f703685-4285-43da-a074-362a513a32b1","Type":"ContainerStarted","Data":"9a3db0c097f152236150c7bd626531d86cbb7d345c5c349b11c54314a25a5324"}
Jul 14 11:40:14.885897 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:40:14.885483    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5qmr8" event={"ID":"4f703685-4285-43da-a074-362a513a32b1","Type":"ContainerStarted","Data":"8d8246187bb14677f7d1fc6c23c7d9258eeb4c8e647264d9309116e6cb18cbe3"}
Jul 14 11:40:14.885897 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:40:14.885569    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-5qmr8"
Jul 14 11:40:14.904884 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:40:14.904810    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-5qmr8" podStartSLOduration=129.902773012 podStartE2EDuration="2m11.904791778s" totalImagesPullingTime="2.002018766s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 11:38:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 11:40:14.904541313 +0000 UTC m=+166.136350608" watchObservedRunningTime="2026-07-14 11:40:14.904791778 +0000 UTC m=+166.136601087"
Jul 14 11:40:24.890895 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:40:24.890816    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-5qmr8"
Jul 14 11:40:27.039395 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:40:27.039354    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:40:28.765283 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:40:28.765243    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:40:28.842149 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:40:28.842110    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:40:34.057482 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:40:34.057451    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:40:34.128704 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:40:34.128665    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:40:52.493424 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:40:52.493382    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:40:57.592136 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:40:57.592096    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:40:57.689625 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:40:57.689588    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:41:09.500625 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:41:09.500588    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:41:09.601838 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:41:09.601805    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:41:13.761242 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:41:13.761202    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:41:13.865680 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:41:13.865639    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:41:14.001287 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:41:14.001250    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:41:14.137186 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:41:14.137091    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:41:14.737882 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:41:14.737851    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:41:24.701600 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:41:24.701546    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:41:24.822738 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:41:24.822704    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:41:54.255138 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:41:54.255042    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:41:54.352536 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:41:54.352505    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:41:55.964690 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:41:55.964658    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:41:56.038938 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:41:56.038907    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:42:01.889020 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:42:01.888968    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:42:02.033967 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:42:02.033929    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:42:21.743020 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:42:21.742978    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:42:22.507348 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:42:22.507310    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:42:25.546656 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:42:25.546620    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:42:25.607294 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:42:25.607257    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:42:27.104338 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:42:27.104305    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:42:27.197235 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:42:27.197202    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:42:27.838395 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:42:27.838359    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:42:27.948209 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:42:27.948171    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:42:29.055047 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:42:29.055016    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cp57b_62c98a11-ece4-4497-99e7-0840a7626a56/ovn-acl-logging/0.log"
Jul 14 11:42:29.059211 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:42:29.059188    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cp57b_62c98a11-ece4-4497-99e7-0840a7626a56/ovn-acl-logging/0.log"
Jul 14 11:42:29.061160 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:42:29.061142    2551 kubelet.go:1648] "Image garbage collection succeeded"
Jul 14 11:42:29.673994 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:42:29.673956    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:42:30.296244 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:42:30.296211    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:42:32.342522 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:42:32.342491    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:42:32.442191 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:42:32.442151    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:42:39.072733 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:42:39.072701    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:42:40.602805 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:42:40.602770    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-twvb7"]
Jul 14 11:42:40.605719 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:42:40.605702    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-twvb7"
Jul 14 11:42:40.608589 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:42:40.608553    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-kp2dz\""
Jul 14 11:42:40.608709 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:42:40.608597    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 14 11:42:40.608709 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:42:40.608604    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 14 11:42:40.615259 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:42:40.615239    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-twvb7"]
Jul 14 11:42:40.666525 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:42:40.666500    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e8d0442f-603f-4a5e-aa73-449426463629-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-twvb7\" (UID: \"e8d0442f-603f-4a5e-aa73-449426463629\") " pod="cert-manager/cert-manager-webhook-6486ffd997-twvb7"
Jul 14 11:42:40.666657 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:42:40.666554    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svlmj\" (UniqueName: \"kubernetes.io/projected/e8d0442f-603f-4a5e-aa73-449426463629-kube-api-access-svlmj\") pod \"cert-manager-webhook-6486ffd997-twvb7\" (UID: \"e8d0442f-603f-4a5e-aa73-449426463629\") " pod="cert-manager/cert-manager-webhook-6486ffd997-twvb7"
Jul 14 11:42:40.767826 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:42:40.767797    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svlmj\" (UniqueName: \"kubernetes.io/projected/e8d0442f-603f-4a5e-aa73-449426463629-kube-api-access-svlmj\") pod \"cert-manager-webhook-6486ffd997-twvb7\" (UID: \"e8d0442f-603f-4a5e-aa73-449426463629\") " pod="cert-manager/cert-manager-webhook-6486ffd997-twvb7"
Jul 14 11:42:40.767949 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:42:40.767833    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e8d0442f-603f-4a5e-aa73-449426463629-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-twvb7\" (UID: \"e8d0442f-603f-4a5e-aa73-449426463629\") " pod="cert-manager/cert-manager-webhook-6486ffd997-twvb7"
Jul 14 11:42:40.777281 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:42:40.777257    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e8d0442f-603f-4a5e-aa73-449426463629-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-twvb7\" (UID: \"e8d0442f-603f-4a5e-aa73-449426463629\") " pod="cert-manager/cert-manager-webhook-6486ffd997-twvb7"
Jul 14 11:42:40.777374 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:42:40.777307    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-svlmj\" (UniqueName: \"kubernetes.io/projected/e8d0442f-603f-4a5e-aa73-449426463629-kube-api-access-svlmj\") pod \"cert-manager-webhook-6486ffd997-twvb7\" (UID: \"e8d0442f-603f-4a5e-aa73-449426463629\") " pod="cert-manager/cert-manager-webhook-6486ffd997-twvb7"
Jul 14 11:42:40.915522 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:42:40.915456    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-twvb7"
Jul 14 11:42:41.044661 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:42:41.044548    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-twvb7"]
Jul 14 11:42:41.047103 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:42:41.047069    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode8d0442f_603f_4a5e_aa73_449426463629.slice/crio-6ad18781709d7b76138d359a862fd5bc9af34517ec890b77b9322edeb01ca765 WatchSource:0}: Error finding container 6ad18781709d7b76138d359a862fd5bc9af34517ec890b77b9322edeb01ca765: Status 404 returned error can't find the container with id 6ad18781709d7b76138d359a862fd5bc9af34517ec890b77b9322edeb01ca765
Jul 14 11:42:41.049049 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:42:41.049030    2551 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 14 11:42:41.362380 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:42:41.362348    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-twvb7" event={"ID":"e8d0442f-603f-4a5e-aa73-449426463629","Type":"ContainerStarted","Data":"6ad18781709d7b76138d359a862fd5bc9af34517ec890b77b9322edeb01ca765"}
Jul 14 11:42:44.318454 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:42:44.318365    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:42:44.374346 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:42:44.374319    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-twvb7" event={"ID":"e8d0442f-603f-4a5e-aa73-449426463629","Type":"ContainerStarted","Data":"aaae5f5d726d87960fe24fd9864adb348be281e6067c6bd9d711ab70ef5a6083"}
Jul 14 11:42:44.374488 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:42:44.374371    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-twvb7"
Jul 14 11:42:44.395644 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:42:44.395598    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-twvb7" podStartSLOduration=1.43278956 podStartE2EDuration="4.395562531s" totalImagesPullingTime="2.962772971s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 11:42:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 11:42:44.394939987 +0000 UTC m=+315.626749284" watchObservedRunningTime="2026-07-14 11:42:44.395562531 +0000 UTC m=+315.627371826"
Jul 14 11:42:44.401683 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:42:44.401659    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:42:48.523189 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:42:48.523159    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-6g8xv"]
Jul 14 11:42:48.526248 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:42:48.526230    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-6g8xv"
Jul 14 11:42:48.528869 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:42:48.528850    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-x4ltn\""
Jul 14 11:42:48.540815 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:42:48.540794    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-6g8xv"]
Jul 14 11:42:48.625507 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:42:48.625481    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b3ed3163-61f8-4e26-8ea8-ca00b42a3b3b-bound-sa-token\") pod \"cert-manager-55b4d954d5-6g8xv\" (UID: \"b3ed3163-61f8-4e26-8ea8-ca00b42a3b3b\") " pod="cert-manager/cert-manager-55b4d954d5-6g8xv"
Jul 14 11:42:48.625624 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:42:48.625520    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9xtr\" (UniqueName: \"kubernetes.io/projected/b3ed3163-61f8-4e26-8ea8-ca00b42a3b3b-kube-api-access-p9xtr\") pod \"cert-manager-55b4d954d5-6g8xv\" (UID: \"b3ed3163-61f8-4e26-8ea8-ca00b42a3b3b\") " pod="cert-manager/cert-manager-55b4d954d5-6g8xv"
Jul 14 11:42:48.726219 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:42:48.726194    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b3ed3163-61f8-4e26-8ea8-ca00b42a3b3b-bound-sa-token\") pod \"cert-manager-55b4d954d5-6g8xv\" (UID: \"b3ed3163-61f8-4e26-8ea8-ca00b42a3b3b\") " pod="cert-manager/cert-manager-55b4d954d5-6g8xv"
Jul 14 11:42:48.726307 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:42:48.726235    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9xtr\" (UniqueName: \"kubernetes.io/projected/b3ed3163-61f8-4e26-8ea8-ca00b42a3b3b-kube-api-access-p9xtr\") pod \"cert-manager-55b4d954d5-6g8xv\" (UID: \"b3ed3163-61f8-4e26-8ea8-ca00b42a3b3b\") " pod="cert-manager/cert-manager-55b4d954d5-6g8xv"
Jul 14 11:42:48.736838 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:42:48.736809    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9xtr\" (UniqueName: \"kubernetes.io/projected/b3ed3163-61f8-4e26-8ea8-ca00b42a3b3b-kube-api-access-p9xtr\") pod \"cert-manager-55b4d954d5-6g8xv\" (UID: \"b3ed3163-61f8-4e26-8ea8-ca00b42a3b3b\") " pod="cert-manager/cert-manager-55b4d954d5-6g8xv"
Jul 14 11:42:48.737009 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:42:48.736988    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b3ed3163-61f8-4e26-8ea8-ca00b42a3b3b-bound-sa-token\") pod \"cert-manager-55b4d954d5-6g8xv\" (UID: \"b3ed3163-61f8-4e26-8ea8-ca00b42a3b3b\") " pod="cert-manager/cert-manager-55b4d954d5-6g8xv"
Jul 14 11:42:48.835128 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:42:48.835070    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-6g8xv"
Jul 14 11:42:48.959150 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:42:48.959121    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-6g8xv"]
Jul 14 11:42:48.962003 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:42:48.961975    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb3ed3163_61f8_4e26_8ea8_ca00b42a3b3b.slice/crio-1c7087b8b6ff60f4c1f4fbc3b9d9432dd6d29510b0add5caca9ab56859f27ad3 WatchSource:0}: Error finding container 1c7087b8b6ff60f4c1f4fbc3b9d9432dd6d29510b0add5caca9ab56859f27ad3: Status 404 returned error can't find the container with id 1c7087b8b6ff60f4c1f4fbc3b9d9432dd6d29510b0add5caca9ab56859f27ad3
Jul 14 11:42:49.391930 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:42:49.391893    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-6g8xv" event={"ID":"b3ed3163-61f8-4e26-8ea8-ca00b42a3b3b","Type":"ContainerStarted","Data":"4eaed2045f76847ae8ed25c975621445dbaffb1754497e025bff956c38a11051"}
Jul 14 11:42:49.391930 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:42:49.391935    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-6g8xv" event={"ID":"b3ed3163-61f8-4e26-8ea8-ca00b42a3b3b","Type":"ContainerStarted","Data":"1c7087b8b6ff60f4c1f4fbc3b9d9432dd6d29510b0add5caca9ab56859f27ad3"}
Jul 14 11:42:49.418452 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:42:49.418396    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-6g8xv" podStartSLOduration=1.418381983 podStartE2EDuration="1.418381983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 11:42:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 11:42:49.416706803 +0000 UTC m=+320.648516119" watchObservedRunningTime="2026-07-14 11:42:49.418381983 +0000 UTC m=+320.650191278"
Jul 14 11:42:50.380091 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:42:50.380061    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-twvb7"
Jul 14 11:42:57.290050 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:42:57.290018    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:42:57.404032 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:42:57.403996    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:42:58.443065 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:42:58.443033    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:43:02.657450 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:02.657414    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:43:02.757900 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:02.757846    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:43:04.761013 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:04.760983    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:43:18.985589 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:18.985533    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:43:20.070605 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:20.070556    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:43:20.527766 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:20.527740    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:43:20.661300 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:20.661270    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:43:21.442237 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:21.442203    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:43:21.726216 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:21.726116    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:43:21.838023 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:21.837989    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:43:21.919056 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:21.919028    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6c944d8cbf-xl5vp"]
Jul 14 11:43:21.922058 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:21.922041    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6c944d8cbf-xl5vp"
Jul 14 11:43:21.925096 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:21.925069    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 14 11:43:21.925230 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:21.925211    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 14 11:43:21.925292 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:21.925229    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 14 11:43:21.925351 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:21.925289    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-rfn8t\""
Jul 14 11:43:21.925351 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:21.925300    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 14 11:43:21.942021 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:21.941997    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6c944d8cbf-xl5vp"]
Jul 14 11:43:22.053900 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:22.053830    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/13561ca4-eb2a-4cc6-9abd-3f8ce624cbc9-manifests\") pod \"opendatahub-operator-controller-manager-6c944d8cbf-xl5vp\" (UID: \"13561ca4-eb2a-4cc6-9abd-3f8ce624cbc9\") " pod="opendatahub/opendatahub-operator-controller-manager-6c944d8cbf-xl5vp"
Jul 14 11:43:22.053900 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:22.053866    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/13561ca4-eb2a-4cc6-9abd-3f8ce624cbc9-webhook-cert\") pod \"opendatahub-operator-controller-manager-6c944d8cbf-xl5vp\" (UID: \"13561ca4-eb2a-4cc6-9abd-3f8ce624cbc9\") " pod="opendatahub/opendatahub-operator-controller-manager-6c944d8cbf-xl5vp"
Jul 14 11:43:22.054057 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:22.053927    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/13561ca4-eb2a-4cc6-9abd-3f8ce624cbc9-tmp\") pod \"opendatahub-operator-controller-manager-6c944d8cbf-xl5vp\" (UID: \"13561ca4-eb2a-4cc6-9abd-3f8ce624cbc9\") " pod="opendatahub/opendatahub-operator-controller-manager-6c944d8cbf-xl5vp"
Jul 14 11:43:22.054057 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:22.053948    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csn8t\" (UniqueName: \"kubernetes.io/projected/13561ca4-eb2a-4cc6-9abd-3f8ce624cbc9-kube-api-access-csn8t\") pod \"opendatahub-operator-controller-manager-6c944d8cbf-xl5vp\" (UID: \"13561ca4-eb2a-4cc6-9abd-3f8ce624cbc9\") " pod="opendatahub/opendatahub-operator-controller-manager-6c944d8cbf-xl5vp"
Jul 14 11:43:22.054057 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:22.053976    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/13561ca4-eb2a-4cc6-9abd-3f8ce624cbc9-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6c944d8cbf-xl5vp\" (UID: \"13561ca4-eb2a-4cc6-9abd-3f8ce624cbc9\") " pod="opendatahub/opendatahub-operator-controller-manager-6c944d8cbf-xl5vp"
Jul 14 11:43:22.105433 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:22.105403    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:43:22.154644 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:22.154606    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/13561ca4-eb2a-4cc6-9abd-3f8ce624cbc9-tmp\") pod \"opendatahub-operator-controller-manager-6c944d8cbf-xl5vp\" (UID: \"13561ca4-eb2a-4cc6-9abd-3f8ce624cbc9\") " pod="opendatahub/opendatahub-operator-controller-manager-6c944d8cbf-xl5vp"
Jul 14 11:43:22.154644 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:22.154644    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csn8t\" (UniqueName: \"kubernetes.io/projected/13561ca4-eb2a-4cc6-9abd-3f8ce624cbc9-kube-api-access-csn8t\") pod \"opendatahub-operator-controller-manager-6c944d8cbf-xl5vp\" (UID: \"13561ca4-eb2a-4cc6-9abd-3f8ce624cbc9\") " pod="opendatahub/opendatahub-operator-controller-manager-6c944d8cbf-xl5vp"
Jul 14 11:43:22.154878 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:22.154770    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/13561ca4-eb2a-4cc6-9abd-3f8ce624cbc9-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6c944d8cbf-xl5vp\" (UID: \"13561ca4-eb2a-4cc6-9abd-3f8ce624cbc9\") " pod="opendatahub/opendatahub-operator-controller-manager-6c944d8cbf-xl5vp"
Jul 14 11:43:22.154878 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:22.154859    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/13561ca4-eb2a-4cc6-9abd-3f8ce624cbc9-manifests\") pod \"opendatahub-operator-controller-manager-6c944d8cbf-xl5vp\" (UID: \"13561ca4-eb2a-4cc6-9abd-3f8ce624cbc9\") " pod="opendatahub/opendatahub-operator-controller-manager-6c944d8cbf-xl5vp"
Jul 14 11:43:22.154993 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:22.154900    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/13561ca4-eb2a-4cc6-9abd-3f8ce624cbc9-webhook-cert\") pod \"opendatahub-operator-controller-manager-6c944d8cbf-xl5vp\" (UID: \"13561ca4-eb2a-4cc6-9abd-3f8ce624cbc9\") " pod="opendatahub/opendatahub-operator-controller-manager-6c944d8cbf-xl5vp"
Jul 14 11:43:22.155200 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:22.155174    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/13561ca4-eb2a-4cc6-9abd-3f8ce624cbc9-manifests\") pod \"opendatahub-operator-controller-manager-6c944d8cbf-xl5vp\" (UID: \"13561ca4-eb2a-4cc6-9abd-3f8ce624cbc9\") " pod="opendatahub/opendatahub-operator-controller-manager-6c944d8cbf-xl5vp"
Jul 14 11:43:22.157032 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:22.157012    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/13561ca4-eb2a-4cc6-9abd-3f8ce624cbc9-tmp\") pod \"opendatahub-operator-controller-manager-6c944d8cbf-xl5vp\" (UID: \"13561ca4-eb2a-4cc6-9abd-3f8ce624cbc9\") " pod="opendatahub/opendatahub-operator-controller-manager-6c944d8cbf-xl5vp"
Jul 14 11:43:22.157170 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:22.157155    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/13561ca4-eb2a-4cc6-9abd-3f8ce624cbc9-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6c944d8cbf-xl5vp\" (UID: \"13561ca4-eb2a-4cc6-9abd-3f8ce624cbc9\") " pod="opendatahub/opendatahub-operator-controller-manager-6c944d8cbf-xl5vp"
Jul 14 11:43:22.157336 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:22.157314    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/13561ca4-eb2a-4cc6-9abd-3f8ce624cbc9-webhook-cert\") pod \"opendatahub-operator-controller-manager-6c944d8cbf-xl5vp\" (UID: \"13561ca4-eb2a-4cc6-9abd-3f8ce624cbc9\") " pod="opendatahub/opendatahub-operator-controller-manager-6c944d8cbf-xl5vp"
Jul 14 11:43:22.165343 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:22.165322    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-csn8t\" (UniqueName: \"kubernetes.io/projected/13561ca4-eb2a-4cc6-9abd-3f8ce624cbc9-kube-api-access-csn8t\") pod \"opendatahub-operator-controller-manager-6c944d8cbf-xl5vp\" (UID: \"13561ca4-eb2a-4cc6-9abd-3f8ce624cbc9\") " pod="opendatahub/opendatahub-operator-controller-manager-6c944d8cbf-xl5vp"
Jul 14 11:43:22.219014 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:22.218983    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:43:22.232881 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:22.232856    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6c944d8cbf-xl5vp"
Jul 14 11:43:22.328965 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:22.328913    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:43:22.364518 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:22.364491    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6c944d8cbf-xl5vp"]
Jul 14 11:43:22.367927 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:43:22.367903    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13561ca4_eb2a_4cc6_9abd_3f8ce624cbc9.slice/crio-a780563941cba2144ad07a4d3fd7967f211cd61f04cf8847bd9b42f1216bb8e4 WatchSource:0}: Error finding container a780563941cba2144ad07a4d3fd7967f211cd61f04cf8847bd9b42f1216bb8e4: Status 404 returned error can't find the container with id a780563941cba2144ad07a4d3fd7967f211cd61f04cf8847bd9b42f1216bb8e4
Jul 14 11:43:22.417023 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:22.416982    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:43:22.507650 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:22.507611    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6c944d8cbf-xl5vp" event={"ID":"13561ca4-eb2a-4cc6-9abd-3f8ce624cbc9","Type":"ContainerStarted","Data":"a780563941cba2144ad07a4d3fd7967f211cd61f04cf8847bd9b42f1216bb8e4"}
Jul 14 11:43:22.516010 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:22.515987    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:43:22.618548 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:22.618443    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:43:22.720604 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:22.720555    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:43:25.520905 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:25.520821    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6c944d8cbf-xl5vp" event={"ID":"13561ca4-eb2a-4cc6-9abd-3f8ce624cbc9","Type":"ContainerStarted","Data":"e5f01bced4bb18f9f6956906f2a1ae823a766dcfba688b642f837471bd5d85af"}
Jul 14 11:43:26.526160 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:26.526124    2551 generic.go:356] "Generic (PLEG): container finished" podID="13561ca4-eb2a-4cc6-9abd-3f8ce624cbc9" containerID="e5f01bced4bb18f9f6956906f2a1ae823a766dcfba688b642f837471bd5d85af" exitCode=0
Jul 14 11:43:26.526160 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:26.526167    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6c944d8cbf-xl5vp" event={"ID":"13561ca4-eb2a-4cc6-9abd-3f8ce624cbc9","Type":"ContainerDied","Data":"e5f01bced4bb18f9f6956906f2a1ae823a766dcfba688b642f837471bd5d85af"}
Jul 14 11:43:27.536039 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:27.536002    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6c944d8cbf-xl5vp" event={"ID":"13561ca4-eb2a-4cc6-9abd-3f8ce624cbc9","Type":"ContainerStarted","Data":"8ae12eb2f4ae149c3eedc99006a9373b82a32c10e04d32fcb5fd1ff83a9f7943"}
Jul 14 11:43:27.536480 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:27.536143    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6c944d8cbf-xl5vp"
Jul 14 11:43:27.574135 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:27.574079    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6c944d8cbf-xl5vp" podStartSLOduration=2.241397831 podStartE2EDuration="6.574062613s" totalImagesPullingTime="3.300922304s" totalInitContainerRuntime="1.031742478s" isStatefulPod=false podCreationTimestamp="2026-07-14 11:43:21 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 11:43:27.571725428 +0000 UTC m=+358.803534743" watchObservedRunningTime="2026-07-14 11:43:27.574062613 +0000 UTC m=+358.805871908"
Jul 14 11:43:33.644633 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:33.644589    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:43:33.744497 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:33.744458    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:43:38.543279 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:38.543249    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6c944d8cbf-xl5vp"
Jul 14 11:43:41.446033 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:41.446000    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:43:41.552983 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:41.552929    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:43:41.835060 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:41.834972    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:43:41.994663 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:41.994623    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7d55c74c4b-mhfwz"]
Jul 14 11:43:42.002773 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:42.002750    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7d55c74c4b-mhfwz"
Jul 14 11:43:42.005955 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:42.005929    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 14 11:43:42.006090 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:42.005932    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 14 11:43:42.006239 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:42.006224    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 14 11:43:42.007114 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:42.007097    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 14 11:43:42.007215 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:42.007101    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-cdrws\""
Jul 14 11:43:42.008796 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:42.008778    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7d55c74c4b-mhfwz"]
Jul 14 11:43:42.030170 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:42.030145    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:43:42.108080 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:42.107989    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rl5bn\" (UniqueName: \"kubernetes.io/projected/341d716b-2dda-4d65-83d1-a2b4e64ae139-kube-api-access-rl5bn\") pod \"kube-auth-proxy-7d55c74c4b-mhfwz\" (UID: \"341d716b-2dda-4d65-83d1-a2b4e64ae139\") " pod="openshift-ingress/kube-auth-proxy-7d55c74c4b-mhfwz"
Jul 14 11:43:42.108080 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:42.108057    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/341d716b-2dda-4d65-83d1-a2b4e64ae139-tmp\") pod \"kube-auth-proxy-7d55c74c4b-mhfwz\" (UID: \"341d716b-2dda-4d65-83d1-a2b4e64ae139\") " pod="openshift-ingress/kube-auth-proxy-7d55c74c4b-mhfwz"
Jul 14 11:43:42.108267 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:42.108112    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/341d716b-2dda-4d65-83d1-a2b4e64ae139-tls-certs\") pod \"kube-auth-proxy-7d55c74c4b-mhfwz\" (UID: \"341d716b-2dda-4d65-83d1-a2b4e64ae139\") " pod="openshift-ingress/kube-auth-proxy-7d55c74c4b-mhfwz"
Jul 14 11:43:42.208731 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:42.208691    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/341d716b-2dda-4d65-83d1-a2b4e64ae139-tmp\") pod \"kube-auth-proxy-7d55c74c4b-mhfwz\" (UID: \"341d716b-2dda-4d65-83d1-a2b4e64ae139\") " pod="openshift-ingress/kube-auth-proxy-7d55c74c4b-mhfwz"
Jul 14 11:43:42.208731 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:42.208732    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/341d716b-2dda-4d65-83d1-a2b4e64ae139-tls-certs\") pod \"kube-auth-proxy-7d55c74c4b-mhfwz\" (UID: \"341d716b-2dda-4d65-83d1-a2b4e64ae139\") " pod="openshift-ingress/kube-auth-proxy-7d55c74c4b-mhfwz"
Jul 14 11:43:42.208985 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:42.208776    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rl5bn\" (UniqueName: \"kubernetes.io/projected/341d716b-2dda-4d65-83d1-a2b4e64ae139-kube-api-access-rl5bn\") pod \"kube-auth-proxy-7d55c74c4b-mhfwz\" (UID: \"341d716b-2dda-4d65-83d1-a2b4e64ae139\") " pod="openshift-ingress/kube-auth-proxy-7d55c74c4b-mhfwz"
Jul 14 11:43:42.211058 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:42.211034    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/341d716b-2dda-4d65-83d1-a2b4e64ae139-tmp\") pod \"kube-auth-proxy-7d55c74c4b-mhfwz\" (UID: \"341d716b-2dda-4d65-83d1-a2b4e64ae139\") " pod="openshift-ingress/kube-auth-proxy-7d55c74c4b-mhfwz"
Jul 14 11:43:42.211202 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:42.211180    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/341d716b-2dda-4d65-83d1-a2b4e64ae139-tls-certs\") pod \"kube-auth-proxy-7d55c74c4b-mhfwz\" (UID: \"341d716b-2dda-4d65-83d1-a2b4e64ae139\") " pod="openshift-ingress/kube-auth-proxy-7d55c74c4b-mhfwz"
Jul 14 11:43:42.236434 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:42.236398    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rl5bn\" (UniqueName: \"kubernetes.io/projected/341d716b-2dda-4d65-83d1-a2b4e64ae139-kube-api-access-rl5bn\") pod \"kube-auth-proxy-7d55c74c4b-mhfwz\" (UID: \"341d716b-2dda-4d65-83d1-a2b4e64ae139\") " pod="openshift-ingress/kube-auth-proxy-7d55c74c4b-mhfwz"
Jul 14 11:43:42.326127 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:42.326083    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7d55c74c4b-mhfwz"
Jul 14 11:43:42.430009 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:42.429952    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-b9lvb"]
Jul 14 11:43:42.433860 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:42.433831    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-b9lvb"
Jul 14 11:43:42.436703 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:42.436673    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Jul 14 11:43:42.436854 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:42.436726    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-2vd46\""
Jul 14 11:43:42.448845 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:42.448815    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-b9lvb"]
Jul 14 11:43:42.466787 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:42.466749    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7d55c74c4b-mhfwz"]
Jul 14 11:43:42.469983 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:43:42.469946    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod341d716b_2dda_4d65_83d1_a2b4e64ae139.slice/crio-5ecda278ee43e82051f1c0eb41a19477c7282b579de9cf5962bc8ee3ebbcf55e WatchSource:0}: Error finding container 5ecda278ee43e82051f1c0eb41a19477c7282b579de9cf5962bc8ee3ebbcf55e: Status 404 returned error can't find the container with id 5ecda278ee43e82051f1c0eb41a19477c7282b579de9cf5962bc8ee3ebbcf55e
Jul 14 11:43:42.475801 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:42.475772    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:43:42.511794 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:42.511761    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zx4s6\" (UniqueName: \"kubernetes.io/projected/702307ef-b102-4d06-8246-c81e31a73276-kube-api-access-zx4s6\") pod \"odh-model-controller-7998c67dc4-b9lvb\" (UID: \"702307ef-b102-4d06-8246-c81e31a73276\") " pod="opendatahub/odh-model-controller-7998c67dc4-b9lvb"
Jul 14 11:43:42.511975 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:42.511808    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/702307ef-b102-4d06-8246-c81e31a73276-cert\") pod \"odh-model-controller-7998c67dc4-b9lvb\" (UID: \"702307ef-b102-4d06-8246-c81e31a73276\") " pod="opendatahub/odh-model-controller-7998c67dc4-b9lvb"
Jul 14 11:43:42.579277 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:42.579241    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:43:42.589052 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:42.589023    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7d55c74c4b-mhfwz" event={"ID":"341d716b-2dda-4d65-83d1-a2b4e64ae139","Type":"ContainerStarted","Data":"5ecda278ee43e82051f1c0eb41a19477c7282b579de9cf5962bc8ee3ebbcf55e"}
Jul 14 11:43:42.612510 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:42.612471    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zx4s6\" (UniqueName: \"kubernetes.io/projected/702307ef-b102-4d06-8246-c81e31a73276-kube-api-access-zx4s6\") pod \"odh-model-controller-7998c67dc4-b9lvb\" (UID: \"702307ef-b102-4d06-8246-c81e31a73276\") " pod="opendatahub/odh-model-controller-7998c67dc4-b9lvb"
Jul 14 11:43:42.612712 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:42.612517    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/702307ef-b102-4d06-8246-c81e31a73276-cert\") pod \"odh-model-controller-7998c67dc4-b9lvb\" (UID: \"702307ef-b102-4d06-8246-c81e31a73276\") " pod="opendatahub/odh-model-controller-7998c67dc4-b9lvb"
Jul 14 11:43:42.615018 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:42.614997    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/702307ef-b102-4d06-8246-c81e31a73276-cert\") pod \"odh-model-controller-7998c67dc4-b9lvb\" (UID: \"702307ef-b102-4d06-8246-c81e31a73276\") " pod="opendatahub/odh-model-controller-7998c67dc4-b9lvb"
Jul 14 11:43:42.622798 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:42.622776    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zx4s6\" (UniqueName: \"kubernetes.io/projected/702307ef-b102-4d06-8246-c81e31a73276-kube-api-access-zx4s6\") pod \"odh-model-controller-7998c67dc4-b9lvb\" (UID: \"702307ef-b102-4d06-8246-c81e31a73276\") " pod="opendatahub/odh-model-controller-7998c67dc4-b9lvb"
Jul 14 11:43:42.679288 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:42.679154    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:43:42.745307 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:42.745258    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-b9lvb"
Jul 14 11:43:42.777559 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:42.777527    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:43:42.902798 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:42.902776    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-b9lvb"]
Jul 14 11:43:42.905047 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:43:42.905022    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod702307ef_b102_4d06_8246_c81e31a73276.slice/crio-f74dcdbb01602a6fadf3d45d1f7da7fe457de20c043fb06ffde08ad6daf6d031 WatchSource:0}: Error finding container f74dcdbb01602a6fadf3d45d1f7da7fe457de20c043fb06ffde08ad6daf6d031: Status 404 returned error can't find the container with id f74dcdbb01602a6fadf3d45d1f7da7fe457de20c043fb06ffde08ad6daf6d031
Jul 14 11:43:43.289193 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:43.289158    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-ggvrl"]
Jul 14 11:43:43.292481 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:43.292446    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:43:43.293318 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:43.293101    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-r8km7"]
Jul 14 11:43:43.293881 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:43.293688    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-ggvrl"
Jul 14 11:43:43.296450 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:43.296426    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-r8km7"
Jul 14 11:43:43.296598 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:43.296485    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-svmsv\""
Jul 14 11:43:43.296954 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:43.296846    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Jul 14 11:43:43.299785 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:43.299298    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 14 11:43:43.299785 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:43.299312    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 14 11:43:43.299785 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:43.299365    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-jkzwd\""
Jul 14 11:43:43.299785 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:43.299599    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 14 11:43:43.300644 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:43.300544    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 14 11:43:43.305789 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:43.305710    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-ggvrl"]
Jul 14 11:43:43.316230 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:43.315470    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-r8km7"]
Jul 14 11:43:43.391074 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:43.391034    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:43:43.418946 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:43.418911    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dt5st\" (UniqueName: \"kubernetes.io/projected/fdbe4ef8-e41a-477b-baec-ce4839523ef3-kube-api-access-dt5st\") pod \"istiod-openshift-gateway-7574459656-r8km7\" (UID: \"fdbe4ef8-e41a-477b-baec-ce4839523ef3\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-r8km7"
Jul 14 11:43:43.419161 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:43.418967    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fdbe4ef8-e41a-477b-baec-ce4839523ef3-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-r8km7\" (UID: \"fdbe4ef8-e41a-477b-baec-ce4839523ef3\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-r8km7"
Jul 14 11:43:43.419161 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:43.418990    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdbe4ef8-e41a-477b-baec-ce4839523ef3-istio-token\") pod \"istiod-openshift-gateway-7574459656-r8km7\" (UID: \"fdbe4ef8-e41a-477b-baec-ce4839523ef3\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-r8km7"
Jul 14 11:43:43.419161 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:43.419065    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdbe4ef8-e41a-477b-baec-ce4839523ef3-local-certs\") pod \"istiod-openshift-gateway-7574459656-r8km7\" (UID: \"fdbe4ef8-e41a-477b-baec-ce4839523ef3\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-r8km7"
Jul 14 11:43:43.419161 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:43.419118    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc2f63ce-780c-454a-a5a1-4827c3f08fd8-tls-certs\") pod \"model-serving-api-54d89bd5fc-ggvrl\" (UID: \"dc2f63ce-780c-454a-a5a1-4827c3f08fd8\") " pod="opendatahub/model-serving-api-54d89bd5fc-ggvrl"
Jul 14 11:43:43.419161 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:43.419149    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/fdbe4ef8-e41a-477b-baec-ce4839523ef3-cacerts\") pod \"istiod-openshift-gateway-7574459656-r8km7\" (UID: \"fdbe4ef8-e41a-477b-baec-ce4839523ef3\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-r8km7"
Jul 14 11:43:43.419442 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:43.419182    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/fdbe4ef8-e41a-477b-baec-ce4839523ef3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-r8km7\" (UID: \"fdbe4ef8-e41a-477b-baec-ce4839523ef3\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-r8km7"
Jul 14 11:43:43.419442 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:43.419220    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5klhc\" (UniqueName: \"kubernetes.io/projected/dc2f63ce-780c-454a-a5a1-4827c3f08fd8-kube-api-access-5klhc\") pod \"model-serving-api-54d89bd5fc-ggvrl\" (UID: \"dc2f63ce-780c-454a-a5a1-4827c3f08fd8\") " pod="opendatahub/model-serving-api-54d89bd5fc-ggvrl"
Jul 14 11:43:43.419442 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:43.419263    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/fdbe4ef8-e41a-477b-baec-ce4839523ef3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-r8km7\" (UID: \"fdbe4ef8-e41a-477b-baec-ce4839523ef3\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-r8km7"
Jul 14 11:43:43.520136 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:43.520099    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5klhc\" (UniqueName: \"kubernetes.io/projected/dc2f63ce-780c-454a-a5a1-4827c3f08fd8-kube-api-access-5klhc\") pod \"model-serving-api-54d89bd5fc-ggvrl\" (UID: \"dc2f63ce-780c-454a-a5a1-4827c3f08fd8\") " pod="opendatahub/model-serving-api-54d89bd5fc-ggvrl"
Jul 14 11:43:43.520136 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:43.520143    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/fdbe4ef8-e41a-477b-baec-ce4839523ef3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-r8km7\" (UID: \"fdbe4ef8-e41a-477b-baec-ce4839523ef3\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-r8km7"
Jul 14 11:43:43.520709 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:43.520276    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt5st\" (UniqueName: \"kubernetes.io/projected/fdbe4ef8-e41a-477b-baec-ce4839523ef3-kube-api-access-dt5st\") pod \"istiod-openshift-gateway-7574459656-r8km7\" (UID: \"fdbe4ef8-e41a-477b-baec-ce4839523ef3\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-r8km7"
Jul 14 11:43:43.520709 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:43.520351    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fdbe4ef8-e41a-477b-baec-ce4839523ef3-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-r8km7\" (UID: \"fdbe4ef8-e41a-477b-baec-ce4839523ef3\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-r8km7"
Jul 14 11:43:43.520709 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:43.520381    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdbe4ef8-e41a-477b-baec-ce4839523ef3-istio-token\") pod \"istiod-openshift-gateway-7574459656-r8km7\" (UID: \"fdbe4ef8-e41a-477b-baec-ce4839523ef3\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-r8km7"
Jul 14 11:43:43.520709 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:43.520415    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdbe4ef8-e41a-477b-baec-ce4839523ef3-local-certs\") pod \"istiod-openshift-gateway-7574459656-r8km7\" (UID: \"fdbe4ef8-e41a-477b-baec-ce4839523ef3\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-r8km7"
Jul 14 11:43:43.520709 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:43.520474    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc2f63ce-780c-454a-a5a1-4827c3f08fd8-tls-certs\") pod \"model-serving-api-54d89bd5fc-ggvrl\" (UID: \"dc2f63ce-780c-454a-a5a1-4827c3f08fd8\") " pod="opendatahub/model-serving-api-54d89bd5fc-ggvrl"
Jul 14 11:43:43.520709 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:43.520507    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/fdbe4ef8-e41a-477b-baec-ce4839523ef3-cacerts\") pod \"istiod-openshift-gateway-7574459656-r8km7\" (UID: \"fdbe4ef8-e41a-477b-baec-ce4839523ef3\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-r8km7"
Jul 14 11:43:43.520709 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:43.520543    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/fdbe4ef8-e41a-477b-baec-ce4839523ef3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-r8km7\" (UID: \"fdbe4ef8-e41a-477b-baec-ce4839523ef3\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-r8km7"
Jul 14 11:43:43.521304 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:43.521251    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/fdbe4ef8-e41a-477b-baec-ce4839523ef3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-r8km7\" (UID: \"fdbe4ef8-e41a-477b-baec-ce4839523ef3\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-r8km7"
Jul 14 11:43:43.523026 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:43.522983    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/fdbe4ef8-e41a-477b-baec-ce4839523ef3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-r8km7\" (UID: \"fdbe4ef8-e41a-477b-baec-ce4839523ef3\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-r8km7"
Jul 14 11:43:43.523166 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:43.523144    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdbe4ef8-e41a-477b-baec-ce4839523ef3-local-certs\") pod \"istiod-openshift-gateway-7574459656-r8km7\" (UID: \"fdbe4ef8-e41a-477b-baec-ce4839523ef3\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-r8km7"
Jul 14 11:43:43.523469 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:43.523447    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fdbe4ef8-e41a-477b-baec-ce4839523ef3-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-r8km7\" (UID: \"fdbe4ef8-e41a-477b-baec-ce4839523ef3\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-r8km7"
Jul 14 11:43:43.523627 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:43.523551    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc2f63ce-780c-454a-a5a1-4827c3f08fd8-tls-certs\") pod \"model-serving-api-54d89bd5fc-ggvrl\" (UID: \"dc2f63ce-780c-454a-a5a1-4827c3f08fd8\") " pod="opendatahub/model-serving-api-54d89bd5fc-ggvrl"
Jul 14 11:43:43.524190 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:43.524166    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/fdbe4ef8-e41a-477b-baec-ce4839523ef3-cacerts\") pod \"istiod-openshift-gateway-7574459656-r8km7\" (UID: \"fdbe4ef8-e41a-477b-baec-ce4839523ef3\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-r8km7"
Jul 14 11:43:43.531556 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:43.531526    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdbe4ef8-e41a-477b-baec-ce4839523ef3-istio-token\") pod \"istiod-openshift-gateway-7574459656-r8km7\" (UID: \"fdbe4ef8-e41a-477b-baec-ce4839523ef3\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-r8km7"
Jul 14 11:43:43.531723 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:43.531641    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5klhc\" (UniqueName: \"kubernetes.io/projected/dc2f63ce-780c-454a-a5a1-4827c3f08fd8-kube-api-access-5klhc\") pod \"model-serving-api-54d89bd5fc-ggvrl\" (UID: \"dc2f63ce-780c-454a-a5a1-4827c3f08fd8\") " pod="opendatahub/model-serving-api-54d89bd5fc-ggvrl"
Jul 14 11:43:43.531723 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:43.531655    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dt5st\" (UniqueName: \"kubernetes.io/projected/fdbe4ef8-e41a-477b-baec-ce4839523ef3-kube-api-access-dt5st\") pod \"istiod-openshift-gateway-7574459656-r8km7\" (UID: \"fdbe4ef8-e41a-477b-baec-ce4839523ef3\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-r8km7"
Jul 14 11:43:43.589858 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:43.589763    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:43:43.596267 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:43.596228    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-b9lvb" event={"ID":"702307ef-b102-4d06-8246-c81e31a73276","Type":"ContainerStarted","Data":"f74dcdbb01602a6fadf3d45d1f7da7fe457de20c043fb06ffde08ad6daf6d031"}
Jul 14 11:43:43.609894 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:43.609642    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-ggvrl"
Jul 14 11:43:43.619242 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:43.618963    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-r8km7"
Jul 14 11:43:43.857748 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:43.857704    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-r8km7"]
Jul 14 11:43:43.861475 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:43:43.861442    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdbe4ef8_e41a_477b_baec_ce4839523ef3.slice/crio-afb193f8ffcd6669870f6ed1a3daf97ddce19b3a0fa1c99bc5de429523270ba4 WatchSource:0}: Error finding container afb193f8ffcd6669870f6ed1a3daf97ddce19b3a0fa1c99bc5de429523270ba4: Status 404 returned error can't find the container with id afb193f8ffcd6669870f6ed1a3daf97ddce19b3a0fa1c99bc5de429523270ba4
Jul 14 11:43:43.898272 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:43.897861    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-ggvrl"]
Jul 14 11:43:43.899271 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:43:43.899240    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc2f63ce_780c_454a_a5a1_4827c3f08fd8.slice/crio-780db15c06f74c70baa9e06e1cb5e5bd6699ab9cb3a6513e606e46ec926c8a67 WatchSource:0}: Error finding container 780db15c06f74c70baa9e06e1cb5e5bd6699ab9cb3a6513e606e46ec926c8a67: Status 404 returned error can't find the container with id 780db15c06f74c70baa9e06e1cb5e5bd6699ab9cb3a6513e606e46ec926c8a67
Jul 14 11:43:44.536684 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:44.536627    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:43:44.602249 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:44.602209    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-ggvrl" event={"ID":"dc2f63ce-780c-454a-a5a1-4827c3f08fd8","Type":"ContainerStarted","Data":"780db15c06f74c70baa9e06e1cb5e5bd6699ab9cb3a6513e606e46ec926c8a67"}
Jul 14 11:43:44.603683 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:44.603614    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-r8km7" event={"ID":"fdbe4ef8-e41a-477b-baec-ce4839523ef3","Type":"ContainerStarted","Data":"afb193f8ffcd6669870f6ed1a3daf97ddce19b3a0fa1c99bc5de429523270ba4"}
Jul 14 11:43:48.369553 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:48.369511    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 14 11:43:48.369968 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:48.369611    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 14 11:43:48.621300 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:48.621199    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7d55c74c4b-mhfwz" event={"ID":"341d716b-2dda-4d65-83d1-a2b4e64ae139","Type":"ContainerStarted","Data":"a27c04fdeee2e5feae229422b97c5bbc0ffd72b08c04671f422e2744a65fabb9"}
Jul 14 11:43:48.622735 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:48.622707    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-b9lvb" event={"ID":"702307ef-b102-4d06-8246-c81e31a73276","Type":"ContainerStarted","Data":"e8292730f57deccb9804439cd0c76295a3e278a5d11455a9a2ca39b3fa202036"}
Jul 14 11:43:48.622867 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:48.622768    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-b9lvb"
Jul 14 11:43:48.624275 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:48.624250    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-r8km7" event={"ID":"fdbe4ef8-e41a-477b-baec-ce4839523ef3","Type":"ContainerStarted","Data":"851ad150c53c0fc6f290548d3703c73f635ea1d2c3ffd33ccada85798ccb44aa"}
Jul 14 11:43:48.624362 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:48.624294    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-r8km7"
Jul 14 11:43:48.625745 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:48.625718    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-ggvrl" event={"ID":"dc2f63ce-780c-454a-a5a1-4827c3f08fd8","Type":"ContainerStarted","Data":"238f927f010d22e1bd4db1e118328f7793edcd76a6912dd1869cb2883a17dfe8"}
Jul 14 11:43:48.625860 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:48.625845    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-ggvrl"
Jul 14 11:43:48.642300 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:48.642242    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7d55c74c4b-mhfwz" podStartSLOduration=1.811285459 podStartE2EDuration="7.642221116s" totalImagesPullingTime="5.830935657s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 11:43:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 11:43:48.640280402 +0000 UTC m=+379.872089709" watchObservedRunningTime="2026-07-14 11:43:48.642221116 +0000 UTC m=+379.874030413"
Jul 14 11:43:48.661497 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:48.661434    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-b9lvb" podStartSLOduration=1.251216307 podStartE2EDuration="6.66141267s" totalImagesPullingTime="5.410196363s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 11:43:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 11:43:48.659935656 +0000 UTC m=+379.891744978" watchObservedRunningTime="2026-07-14 11:43:48.66141267 +0000 UTC m=+379.893221967"
Jul 14 11:43:48.681004 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:48.680783    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-ggvrl" podStartSLOduration=1.276042741 podStartE2EDuration="5.680766114s" totalImagesPullingTime="4.404723373s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 11:43:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 11:43:48.67835159 +0000 UTC m=+379.910160886" watchObservedRunningTime="2026-07-14 11:43:48.680766114 +0000 UTC m=+379.912575412"
Jul 14 11:43:48.704632 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:48.704526    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-r8km7" podStartSLOduration=1.200277515 podStartE2EDuration="5.704500624s" totalImagesPullingTime="4.504223109s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 11:43:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 11:43:48.703824687 +0000 UTC m=+379.935633984" watchObservedRunningTime="2026-07-14 11:43:48.704500624 +0000 UTC m=+379.936309921"
Jul 14 11:43:49.330053 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:49.330015    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:43:49.434711 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:49.434659    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:43:49.632321 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:49.632241    2551 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-r8km7 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 14 11:43:49.632321 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:49.632292    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-r8km7" podUID="fdbe4ef8-e41a-477b-baec-ce4839523ef3" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 14 11:43:50.152612 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:50.152540    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:43:50.329352 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:50.329305    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:43:50.431613 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:50.431490    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:43:52.631930 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:52.631896    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-r8km7"
Jul 14 11:43:54.328721 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:54.328682    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:43:54.392959 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:54.392923    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-gl6pr"]
Jul 14 11:43:54.398697 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:54.398672    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gl6pr"
Jul 14 11:43:54.402667 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:54.402636    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-sfz48\""
Jul 14 11:43:54.403816 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:54.403790    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Jul 14 11:43:54.405316 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:54.405292    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-zqj74"]
Jul 14 11:43:54.410476 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:54.410456    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zqj74"
Jul 14 11:43:54.414877 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:54.414854    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-dd8sb\""
Jul 14 11:43:54.414990 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:54.414897    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Jul 14 11:43:54.416108 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:54.416083    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-gl6pr"]
Jul 14 11:43:54.423332 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:54.423291    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rgqm\" (UniqueName: \"kubernetes.io/projected/1540b9c9-a79b-47b4-8183-ea78634934db-kube-api-access-8rgqm\") pod \"kserve-controller-manager-d6f4b65b8-gl6pr\" (UID: \"1540b9c9-a79b-47b4-8183-ea78634934db\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-gl6pr"
Jul 14 11:43:54.423466 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:54.423349    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ea5ed537-f005-4a9d-8bd2-34dfe38244b1-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-zqj74\" (UID: \"ea5ed537-f005-4a9d-8bd2-34dfe38244b1\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zqj74"
Jul 14 11:43:54.423466 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:54.423426    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkhwz\" (UniqueName: \"kubernetes.io/projected/ea5ed537-f005-4a9d-8bd2-34dfe38244b1-kube-api-access-qkhwz\") pod \"llmisvc-controller-manager-777c6bd7d5-zqj74\" (UID: \"ea5ed537-f005-4a9d-8bd2-34dfe38244b1\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zqj74"
Jul 14 11:43:54.423466 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:54.423455    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1540b9c9-a79b-47b4-8183-ea78634934db-cert\") pod \"kserve-controller-manager-d6f4b65b8-gl6pr\" (UID: \"1540b9c9-a79b-47b4-8183-ea78634934db\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-gl6pr"
Jul 14 11:43:54.424520 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:54.424492    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:43:54.434130 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:54.434102    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-zqj74"]
Jul 14 11:43:54.504017 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:54.503986    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:43:54.524798 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:54.524761    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rgqm\" (UniqueName: \"kubernetes.io/projected/1540b9c9-a79b-47b4-8183-ea78634934db-kube-api-access-8rgqm\") pod \"kserve-controller-manager-d6f4b65b8-gl6pr\" (UID: \"1540b9c9-a79b-47b4-8183-ea78634934db\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-gl6pr"
Jul 14 11:43:54.525004 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:54.524812    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ea5ed537-f005-4a9d-8bd2-34dfe38244b1-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-zqj74\" (UID: \"ea5ed537-f005-4a9d-8bd2-34dfe38244b1\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zqj74"
Jul 14 11:43:54.525004 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:54.524849    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkhwz\" (UniqueName: \"kubernetes.io/projected/ea5ed537-f005-4a9d-8bd2-34dfe38244b1-kube-api-access-qkhwz\") pod \"llmisvc-controller-manager-777c6bd7d5-zqj74\" (UID: \"ea5ed537-f005-4a9d-8bd2-34dfe38244b1\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zqj74"
Jul 14 11:43:54.525004 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:54.524867    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1540b9c9-a79b-47b4-8183-ea78634934db-cert\") pod \"kserve-controller-manager-d6f4b65b8-gl6pr\" (UID: \"1540b9c9-a79b-47b4-8183-ea78634934db\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-gl6pr"
Jul 14 11:43:54.527536 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:54.527510    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1540b9c9-a79b-47b4-8183-ea78634934db-cert\") pod \"kserve-controller-manager-d6f4b65b8-gl6pr\" (UID: \"1540b9c9-a79b-47b4-8183-ea78634934db\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-gl6pr"
Jul 14 11:43:54.527681 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:54.527571    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ea5ed537-f005-4a9d-8bd2-34dfe38244b1-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-zqj74\" (UID: \"ea5ed537-f005-4a9d-8bd2-34dfe38244b1\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zqj74"
Jul 14 11:43:54.537371 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:54.537340    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rgqm\" (UniqueName: \"kubernetes.io/projected/1540b9c9-a79b-47b4-8183-ea78634934db-kube-api-access-8rgqm\") pod \"kserve-controller-manager-d6f4b65b8-gl6pr\" (UID: \"1540b9c9-a79b-47b4-8183-ea78634934db\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-gl6pr"
Jul 14 11:43:54.537925 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:54.537896    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkhwz\" (UniqueName: \"kubernetes.io/projected/ea5ed537-f005-4a9d-8bd2-34dfe38244b1-kube-api-access-qkhwz\") pod \"llmisvc-controller-manager-777c6bd7d5-zqj74\" (UID: \"ea5ed537-f005-4a9d-8bd2-34dfe38244b1\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zqj74"
Jul 14 11:43:54.612662 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:54.612557    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:43:54.709304 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:54.709268    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gl6pr"
Jul 14 11:43:54.721156 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:54.721127    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zqj74"
Jul 14 11:43:54.866344 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:54.866199    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-gl6pr"]
Jul 14 11:43:54.869178 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:43:54.869148    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1540b9c9_a79b_47b4_8183_ea78634934db.slice/crio-4b44332dcf22d566a8529a96042accf2dd1d0e5e08d4b82edf59eebd657d58ab WatchSource:0}: Error finding container 4b44332dcf22d566a8529a96042accf2dd1d0e5e08d4b82edf59eebd657d58ab: Status 404 returned error can't find the container with id 4b44332dcf22d566a8529a96042accf2dd1d0e5e08d4b82edf59eebd657d58ab
Jul 14 11:43:54.886366 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:54.886334    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-zqj74"]
Jul 14 11:43:54.889207 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:43:54.889133    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podea5ed537_f005_4a9d_8bd2_34dfe38244b1.slice/crio-415819ad9828d31abb74dc25ba5f086d315fa6261a31082d04b5eb0de4db04fa WatchSource:0}: Error finding container 415819ad9828d31abb74dc25ba5f086d315fa6261a31082d04b5eb0de4db04fa: Status 404 returned error can't find the container with id 415819ad9828d31abb74dc25ba5f086d315fa6261a31082d04b5eb0de4db04fa
Jul 14 11:43:55.595385 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:55.595349    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:43:55.654416 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:55.654376    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gl6pr" event={"ID":"1540b9c9-a79b-47b4-8183-ea78634934db","Type":"ContainerStarted","Data":"4b44332dcf22d566a8529a96042accf2dd1d0e5e08d4b82edf59eebd657d58ab"}
Jul 14 11:43:55.655470 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:55.655447    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zqj74" event={"ID":"ea5ed537-f005-4a9d-8bd2-34dfe38244b1","Type":"ContainerStarted","Data":"415819ad9828d31abb74dc25ba5f086d315fa6261a31082d04b5eb0de4db04fa"}
Jul 14 11:43:55.697754 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:55.697703    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:43:55.805089 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:55.805027    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:43:58.222563 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:58.222530    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-r8km7"]
Jul 14 11:43:58.222958 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:58.222733    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-r8km7" podUID="fdbe4ef8-e41a-477b-baec-ce4839523ef3" containerName="discovery" containerID="cri-o://851ad150c53c0fc6f290548d3703c73f635ea1d2c3ffd33ccada85798ccb44aa" gracePeriod=30
Jul 14 11:43:58.479216 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:58.479189    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-r8km7"
Jul 14 11:43:58.559380 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:58.559349    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdbe4ef8-e41a-477b-baec-ce4839523ef3-local-certs\") pod \"fdbe4ef8-e41a-477b-baec-ce4839523ef3\" (UID: \"fdbe4ef8-e41a-477b-baec-ce4839523ef3\") "
Jul 14 11:43:58.559568 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:58.559402    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fdbe4ef8-e41a-477b-baec-ce4839523ef3-istio-kubeconfig\") pod \"fdbe4ef8-e41a-477b-baec-ce4839523ef3\" (UID: \"fdbe4ef8-e41a-477b-baec-ce4839523ef3\") "
Jul 14 11:43:58.559568 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:58.559429    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dt5st\" (UniqueName: \"kubernetes.io/projected/fdbe4ef8-e41a-477b-baec-ce4839523ef3-kube-api-access-dt5st\") pod \"fdbe4ef8-e41a-477b-baec-ce4839523ef3\" (UID: \"fdbe4ef8-e41a-477b-baec-ce4839523ef3\") "
Jul 14 11:43:58.559568 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:58.559451    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/fdbe4ef8-e41a-477b-baec-ce4839523ef3-cacerts\") pod \"fdbe4ef8-e41a-477b-baec-ce4839523ef3\" (UID: \"fdbe4ef8-e41a-477b-baec-ce4839523ef3\") "
Jul 14 11:43:58.559568 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:58.559471    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/fdbe4ef8-e41a-477b-baec-ce4839523ef3-istio-csr-ca-configmap\") pod \"fdbe4ef8-e41a-477b-baec-ce4839523ef3\" (UID: \"fdbe4ef8-e41a-477b-baec-ce4839523ef3\") "
Jul 14 11:43:58.559568 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:58.559510    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdbe4ef8-e41a-477b-baec-ce4839523ef3-istio-token\") pod \"fdbe4ef8-e41a-477b-baec-ce4839523ef3\" (UID: \"fdbe4ef8-e41a-477b-baec-ce4839523ef3\") "
Jul 14 11:43:58.559568 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:58.559557    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/fdbe4ef8-e41a-477b-baec-ce4839523ef3-istio-csr-dns-cert\") pod \"fdbe4ef8-e41a-477b-baec-ce4839523ef3\" (UID: \"fdbe4ef8-e41a-477b-baec-ce4839523ef3\") "
Jul 14 11:43:58.559968 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:58.559939    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdbe4ef8-e41a-477b-baec-ce4839523ef3-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "fdbe4ef8-e41a-477b-baec-ce4839523ef3" (UID: "fdbe4ef8-e41a-477b-baec-ce4839523ef3"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 11:43:58.561835 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:58.561802    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdbe4ef8-e41a-477b-baec-ce4839523ef3-local-certs" (OuterVolumeSpecName: "local-certs") pod "fdbe4ef8-e41a-477b-baec-ce4839523ef3" (UID: "fdbe4ef8-e41a-477b-baec-ce4839523ef3"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 11:43:58.561975 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:58.561947    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdbe4ef8-e41a-477b-baec-ce4839523ef3-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "fdbe4ef8-e41a-477b-baec-ce4839523ef3" (UID: "fdbe4ef8-e41a-477b-baec-ce4839523ef3"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 11:43:58.562175 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:58.562147    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdbe4ef8-e41a-477b-baec-ce4839523ef3-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "fdbe4ef8-e41a-477b-baec-ce4839523ef3" (UID: "fdbe4ef8-e41a-477b-baec-ce4839523ef3"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 11:43:58.562175 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:58.562164    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdbe4ef8-e41a-477b-baec-ce4839523ef3-cacerts" (OuterVolumeSpecName: "cacerts") pod "fdbe4ef8-e41a-477b-baec-ce4839523ef3" (UID: "fdbe4ef8-e41a-477b-baec-ce4839523ef3"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 11:43:58.562314 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:58.562178    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdbe4ef8-e41a-477b-baec-ce4839523ef3-kube-api-access-dt5st" (OuterVolumeSpecName: "kube-api-access-dt5st") pod "fdbe4ef8-e41a-477b-baec-ce4839523ef3" (UID: "fdbe4ef8-e41a-477b-baec-ce4839523ef3"). InnerVolumeSpecName "kube-api-access-dt5st". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 11:43:58.562314 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:58.562214    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdbe4ef8-e41a-477b-baec-ce4839523ef3-istio-token" (OuterVolumeSpecName: "istio-token") pod "fdbe4ef8-e41a-477b-baec-ce4839523ef3" (UID: "fdbe4ef8-e41a-477b-baec-ce4839523ef3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 11:43:58.660768 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:58.660726    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dt5st\" (UniqueName: \"kubernetes.io/projected/fdbe4ef8-e41a-477b-baec-ce4839523ef3-kube-api-access-dt5st\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 11:43:58.660768 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:58.660762    2551 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/fdbe4ef8-e41a-477b-baec-ce4839523ef3-cacerts\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 11:43:58.660768 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:58.660773    2551 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/fdbe4ef8-e41a-477b-baec-ce4839523ef3-istio-csr-ca-configmap\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 11:43:58.661011 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:58.660782    2551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdbe4ef8-e41a-477b-baec-ce4839523ef3-istio-token\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 11:43:58.661011 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:58.660790    2551 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/fdbe4ef8-e41a-477b-baec-ce4839523ef3-istio-csr-dns-cert\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 11:43:58.661011 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:58.660798    2551 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdbe4ef8-e41a-477b-baec-ce4839523ef3-local-certs\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 11:43:58.661011 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:58.660806    2551 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fdbe4ef8-e41a-477b-baec-ce4839523ef3-istio-kubeconfig\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 11:43:58.669654 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:58.669619    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gl6pr" event={"ID":"1540b9c9-a79b-47b4-8183-ea78634934db","Type":"ContainerStarted","Data":"c213d368f3fcfeb60a3ffa39d269cce04d0c3cc5503247720cc2959def0800e6"}
Jul 14 11:43:58.669845 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:58.669731    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gl6pr"
Jul 14 11:43:58.671028 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:58.671002    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zqj74" event={"ID":"ea5ed537-f005-4a9d-8bd2-34dfe38244b1","Type":"ContainerStarted","Data":"1a711f1d67e29e807cd306da8e91e452d91152c065b39f37b899a5deeb9cfe5b"}
Jul 14 11:43:58.671161 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:58.671099    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zqj74"
Jul 14 11:43:58.672264 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:58.672241    2551 generic.go:356] "Generic (PLEG): container finished" podID="fdbe4ef8-e41a-477b-baec-ce4839523ef3" containerID="851ad150c53c0fc6f290548d3703c73f635ea1d2c3ffd33ccada85798ccb44aa" exitCode=0
Jul 14 11:43:58.672375 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:58.672272    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-r8km7"
Jul 14 11:43:58.672375 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:58.672290    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-r8km7" event={"ID":"fdbe4ef8-e41a-477b-baec-ce4839523ef3","Type":"ContainerDied","Data":"851ad150c53c0fc6f290548d3703c73f635ea1d2c3ffd33ccada85798ccb44aa"}
Jul 14 11:43:58.672375 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:58.672332    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-r8km7" event={"ID":"fdbe4ef8-e41a-477b-baec-ce4839523ef3","Type":"ContainerDied","Data":"afb193f8ffcd6669870f6ed1a3daf97ddce19b3a0fa1c99bc5de429523270ba4"}
Jul 14 11:43:58.672375 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:58.672358    2551 scope.go:117] "RemoveContainer" containerID="851ad150c53c0fc6f290548d3703c73f635ea1d2c3ffd33ccada85798ccb44aa"
Jul 14 11:43:58.681532 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:58.681511    2551 scope.go:117] "RemoveContainer" containerID="851ad150c53c0fc6f290548d3703c73f635ea1d2c3ffd33ccada85798ccb44aa"
Jul 14 11:43:58.681863 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:43:58.681840    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"851ad150c53c0fc6f290548d3703c73f635ea1d2c3ffd33ccada85798ccb44aa\": container with ID starting with 851ad150c53c0fc6f290548d3703c73f635ea1d2c3ffd33ccada85798ccb44aa not found: ID does not exist" containerID="851ad150c53c0fc6f290548d3703c73f635ea1d2c3ffd33ccada85798ccb44aa"
Jul 14 11:43:58.681966 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:58.681878    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"851ad150c53c0fc6f290548d3703c73f635ea1d2c3ffd33ccada85798ccb44aa"} err="failed to get container status \"851ad150c53c0fc6f290548d3703c73f635ea1d2c3ffd33ccada85798ccb44aa\": rpc error: code = NotFound desc = could not find container \"851ad150c53c0fc6f290548d3703c73f635ea1d2c3ffd33ccada85798ccb44aa\": container with ID starting with 851ad150c53c0fc6f290548d3703c73f635ea1d2c3ffd33ccada85798ccb44aa not found: ID does not exist"
Jul 14 11:43:58.689938 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:58.689893    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gl6pr" podStartSLOduration=1.370707979 podStartE2EDuration="4.689878382s" totalImagesPullingTime="3.319170403s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 11:43:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 11:43:58.689476618 +0000 UTC m=+389.921285933" watchObservedRunningTime="2026-07-14 11:43:58.689878382 +0000 UTC m=+389.921687679"
Jul 14 11:43:58.706142 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:58.706108    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-r8km7"]
Jul 14 11:43:58.709152 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:58.709126    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-r8km7"]
Jul 14 11:43:58.727657 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:58.727488    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zqj74" podStartSLOduration=1.424441894 podStartE2EDuration="4.727471139s" totalImagesPullingTime="3.303029245s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 11:43:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 11:43:58.726489546 +0000 UTC m=+389.958298855" watchObservedRunningTime="2026-07-14 11:43:58.727471139 +0000 UTC m=+389.959280475"
Jul 14 11:43:59.190705 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:59.190668    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fdbe4ef8-e41a-477b-baec-ce4839523ef3" path="/var/lib/kubelet/pods/fdbe4ef8-e41a-477b-baec-ce4839523ef3/volumes"
Jul 14 11:43:59.231427 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:59.231394    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:43:59.335620 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:59.335571    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:43:59.632456 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:59.632374    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-b9lvb"
Jul 14 11:43:59.634644 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:43:59.634622    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-ggvrl"
Jul 14 11:44:29.680494 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:44:29.680464    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zqj74"
Jul 14 11:44:29.683361 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:44:29.683340    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gl6pr"
Jul 14 11:44:34.825343 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:44:34.825312    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:44:42.937603 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:44:42.937550    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:44:43.070663 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:44:43.070619    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:44:43.109327 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:44:43.109296    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-99d4p"]
Jul 14 11:44:43.109572 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:44:43.109561    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdbe4ef8-e41a-477b-baec-ce4839523ef3" containerName="discovery"
Jul 14 11:44:43.109639 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:44:43.109574    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdbe4ef8-e41a-477b-baec-ce4839523ef3" containerName="discovery"
Jul 14 11:44:43.109673 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:44:43.109665    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="fdbe4ef8-e41a-477b-baec-ce4839523ef3" containerName="discovery"
Jul 14 11:44:43.115271 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:44:43.115247    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-99d4p"
Jul 14 11:44:43.118020 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:44:43.117998    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 14 11:44:43.119301 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:44:43.119279    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 14 11:44:43.119431 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:44:43.119278    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-bnzdh\""
Jul 14 11:44:43.125173 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:44:43.125153    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-99d4p"]
Jul 14 11:44:43.178985 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:44:43.178952    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:44:43.214094 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:44:43.213993    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6x9l\" (UniqueName: \"kubernetes.io/projected/340cc9d9-9afb-4f6e-b35d-3c4ceea20c83-kube-api-access-h6x9l\") pod \"kuadrant-operator-catalog-99d4p\" (UID: \"340cc9d9-9afb-4f6e-b35d-3c4ceea20c83\") " pod="kuadrant-system/kuadrant-operator-catalog-99d4p"
Jul 14 11:44:43.314940 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:44:43.314837    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6x9l\" (UniqueName: \"kubernetes.io/projected/340cc9d9-9afb-4f6e-b35d-3c4ceea20c83-kube-api-access-h6x9l\") pod \"kuadrant-operator-catalog-99d4p\" (UID: \"340cc9d9-9afb-4f6e-b35d-3c4ceea20c83\") " pod="kuadrant-system/kuadrant-operator-catalog-99d4p"
Jul 14 11:44:43.325341 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:44:43.325309    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6x9l\" (UniqueName: \"kubernetes.io/projected/340cc9d9-9afb-4f6e-b35d-3c4ceea20c83-kube-api-access-h6x9l\") pod \"kuadrant-operator-catalog-99d4p\" (UID: \"340cc9d9-9afb-4f6e-b35d-3c4ceea20c83\") " pod="kuadrant-system/kuadrant-operator-catalog-99d4p"
Jul 14 11:44:43.425311 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:44:43.425267    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-99d4p"
Jul 14 11:44:43.565092 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:44:43.565059    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-99d4p"]
Jul 14 11:44:43.568564 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:44:43.568524    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod340cc9d9_9afb_4f6e_b35d_3c4ceea20c83.slice/crio-ac1404c0d9bb512dad83af26886464bac0dcc90c7ce99ec4828340e7fabf58ab WatchSource:0}: Error finding container ac1404c0d9bb512dad83af26886464bac0dcc90c7ce99ec4828340e7fabf58ab: Status 404 returned error can't find the container with id ac1404c0d9bb512dad83af26886464bac0dcc90c7ce99ec4828340e7fabf58ab
Jul 14 11:44:43.851144 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:44:43.851048    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-99d4p" event={"ID":"340cc9d9-9afb-4f6e-b35d-3c4ceea20c83","Type":"ContainerStarted","Data":"ac1404c0d9bb512dad83af26886464bac0dcc90c7ce99ec4828340e7fabf58ab"}
Jul 14 11:44:45.865947 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:44:45.865861    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-99d4p" event={"ID":"340cc9d9-9afb-4f6e-b35d-3c4ceea20c83","Type":"ContainerStarted","Data":"50614fe390bb7f6ecdf7412ea08bf83027f368f6218be0ad485d3b7a968a6a1d"}
Jul 14 11:44:45.885732 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:44:45.885669    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-99d4p" podStartSLOduration=0.843853138 podStartE2EDuration="2.885648725s" totalImagesPullingTime="2.041795587s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 11:44:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 11:44:45.8836698 +0000 UTC m=+437.115479098" watchObservedRunningTime="2026-07-14 11:44:45.885648725 +0000 UTC m=+437.117458024"
Jul 14 11:44:53.425743 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:44:53.425656    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-99d4p"
Jul 14 11:44:53.425743 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:44:53.425726    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-99d4p"
Jul 14 11:44:53.449183 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:44:53.449156    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-99d4p"
Jul 14 11:44:53.917118 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:44:53.917088    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-99d4p"
Jul 14 11:45:07.896984 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:07.896905    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:45:07.997759 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:07.997724    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:45:08.154752 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:08.154660    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:45:08.289736 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:08.289705    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:45:11.380236 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:11.380198    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:45:14.014184 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:14.014150    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-m7jvb"]
Jul 14 11:45:14.021644 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:14.021624    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-m7jvb"
Jul 14 11:45:14.024419 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:14.024396    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Jul 14 11:45:14.024516 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:14.024496    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-h8lzc\""
Jul 14 11:45:14.027529 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:14.027502    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-m7jvb"]
Jul 14 11:45:14.054212 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:14.054181    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pw5z\" (UniqueName: \"kubernetes.io/projected/b0531370-9f45-44e1-8e67-f32852cd15a9-kube-api-access-4pw5z\") pod \"dns-operator-controller-manager-864dd78c4c-m7jvb\" (UID: \"b0531370-9f45-44e1-8e67-f32852cd15a9\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-m7jvb"
Jul 14 11:45:14.155204 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:14.155169    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pw5z\" (UniqueName: \"kubernetes.io/projected/b0531370-9f45-44e1-8e67-f32852cd15a9-kube-api-access-4pw5z\") pod \"dns-operator-controller-manager-864dd78c4c-m7jvb\" (UID: \"b0531370-9f45-44e1-8e67-f32852cd15a9\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-m7jvb"
Jul 14 11:45:14.165201 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:14.165168    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pw5z\" (UniqueName: \"kubernetes.io/projected/b0531370-9f45-44e1-8e67-f32852cd15a9-kube-api-access-4pw5z\") pod \"dns-operator-controller-manager-864dd78c4c-m7jvb\" (UID: \"b0531370-9f45-44e1-8e67-f32852cd15a9\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-m7jvb"
Jul 14 11:45:14.332319 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:14.332229    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-m7jvb"
Jul 14 11:45:14.467573 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:14.467538    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-m7jvb"]
Jul 14 11:45:14.469788 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:45:14.469763    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0531370_9f45_44e1_8e67_f32852cd15a9.slice/crio-17e6d9f714e5a10d80506b4630a58b99d26f60064d48df023a8630ee44da803b WatchSource:0}: Error finding container 17e6d9f714e5a10d80506b4630a58b99d26f60064d48df023a8630ee44da803b: Status 404 returned error can't find the container with id 17e6d9f714e5a10d80506b4630a58b99d26f60064d48df023a8630ee44da803b
Jul 14 11:45:14.981097 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:14.981066    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-m7jvb" event={"ID":"b0531370-9f45-44e1-8e67-f32852cd15a9","Type":"ContainerStarted","Data":"17e6d9f714e5a10d80506b4630a58b99d26f60064d48df023a8630ee44da803b"}
Jul 14 11:45:15.626428 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:15.626392    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:45:16.991233 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:16.991198    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-m7jvb" event={"ID":"b0531370-9f45-44e1-8e67-f32852cd15a9","Type":"ContainerStarted","Data":"98a2369217573b1c3ef2a16be3ce2f03af0a245267f71ea8697850fec6579371"}
Jul 14 11:45:16.991657 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:16.991344    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-m7jvb"
Jul 14 11:45:17.019261 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:17.019209    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-m7jvb" podStartSLOduration=2.154314889 podStartE2EDuration="4.019194291s" totalImagesPullingTime="1.864879402s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 11:45:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 11:45:17.017710816 +0000 UTC m=+468.249520113" watchObservedRunningTime="2026-07-14 11:45:17.019194291 +0000 UTC m=+468.251003586"
Jul 14 11:45:17.580773 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:17.580743    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rdgp4"]
Jul 14 11:45:17.584001 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:17.583978    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rdgp4"
Jul 14 11:45:17.586708 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:17.586683    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-2xkqz\""
Jul 14 11:45:17.596591 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:17.596554    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rdgp4"]
Jul 14 11:45:17.682790 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:17.682755    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57ngl\" (UniqueName: \"kubernetes.io/projected/f892a432-e9d2-451b-9a42-fdca3d188149-kube-api-access-57ngl\") pod \"limitador-operator-controller-manager-859d7b78bd-rdgp4\" (UID: \"f892a432-e9d2-451b-9a42-fdca3d188149\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rdgp4"
Jul 14 11:45:17.784092 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:17.784052    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57ngl\" (UniqueName: \"kubernetes.io/projected/f892a432-e9d2-451b-9a42-fdca3d188149-kube-api-access-57ngl\") pod \"limitador-operator-controller-manager-859d7b78bd-rdgp4\" (UID: \"f892a432-e9d2-451b-9a42-fdca3d188149\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rdgp4"
Jul 14 11:45:17.800008 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:17.799973    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-57ngl\" (UniqueName: \"kubernetes.io/projected/f892a432-e9d2-451b-9a42-fdca3d188149-kube-api-access-57ngl\") pod \"limitador-operator-controller-manager-859d7b78bd-rdgp4\" (UID: \"f892a432-e9d2-451b-9a42-fdca3d188149\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rdgp4"
Jul 14 11:45:17.894214 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:17.894117    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rdgp4"
Jul 14 11:45:18.058990 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:18.058951    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rdgp4"]
Jul 14 11:45:18.061751 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:45:18.061724    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf892a432_e9d2_451b_9a42_fdca3d188149.slice/crio-3be6d9deab58a6f7157713c6b4e6a595812140f99e71f4ecfe62564edb2ae1a2 WatchSource:0}: Error finding container 3be6d9deab58a6f7157713c6b4e6a595812140f99e71f4ecfe62564edb2ae1a2: Status 404 returned error can't find the container with id 3be6d9deab58a6f7157713c6b4e6a595812140f99e71f4ecfe62564edb2ae1a2
Jul 14 11:45:18.106618 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:18.106556    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:45:18.390151 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:18.390110    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:45:19.014544 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:19.014506    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rdgp4" event={"ID":"f892a432-e9d2-451b-9a42-fdca3d188149","Type":"ContainerStarted","Data":"3be6d9deab58a6f7157713c6b4e6a595812140f99e71f4ecfe62564edb2ae1a2"}
Jul 14 11:45:19.395763 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:19.395662    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:45:19.683246 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:19.683145    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:45:19.784776 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:19.784714    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:45:21.024748 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:21.024713    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rdgp4" event={"ID":"f892a432-e9d2-451b-9a42-fdca3d188149","Type":"ContainerStarted","Data":"e2b9d9aa638d48077a32329966ca34b9bb5f1f928611c01dd3af971c7cd0f302"}
Jul 14 11:45:21.025171 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:21.024776    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rdgp4"
Jul 14 11:45:21.052648 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:21.052601    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rdgp4" podStartSLOduration=2.156431878 podStartE2EDuration="4.052567909s" totalImagesPullingTime="1.896136031s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 11:45:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 11:45:21.050341222 +0000 UTC m=+472.282150531" watchObservedRunningTime="2026-07-14 11:45:21.052567909 +0000 UTC m=+472.284377205"
Jul 14 11:45:22.593682 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:22.593645    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-726bq"]
Jul 14 11:45:22.596791 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:22.596766    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-726bq"
Jul 14 11:45:22.599562 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:22.599539    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-lx5tn\""
Jul 14 11:45:22.609955 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:22.609929    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-726bq"]
Jul 14 11:45:22.625778 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:22.625750    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8e85fe8f-489a-470b-a22c-e01803a6709a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-726bq\" (UID: \"8e85fe8f-489a-470b-a22c-e01803a6709a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-726bq"
Jul 14 11:45:22.625911 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:22.625793    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jr8dp\" (UniqueName: \"kubernetes.io/projected/8e85fe8f-489a-470b-a22c-e01803a6709a-kube-api-access-jr8dp\") pod \"kuadrant-operator-controller-manager-8684c9ff49-726bq\" (UID: \"8e85fe8f-489a-470b-a22c-e01803a6709a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-726bq"
Jul 14 11:45:22.726688 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:22.726656    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8e85fe8f-489a-470b-a22c-e01803a6709a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-726bq\" (UID: \"8e85fe8f-489a-470b-a22c-e01803a6709a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-726bq"
Jul 14 11:45:22.726859 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:22.726701    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jr8dp\" (UniqueName: \"kubernetes.io/projected/8e85fe8f-489a-470b-a22c-e01803a6709a-kube-api-access-jr8dp\") pod \"kuadrant-operator-controller-manager-8684c9ff49-726bq\" (UID: \"8e85fe8f-489a-470b-a22c-e01803a6709a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-726bq"
Jul 14 11:45:22.727163 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:22.727138    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8e85fe8f-489a-470b-a22c-e01803a6709a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-726bq\" (UID: \"8e85fe8f-489a-470b-a22c-e01803a6709a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-726bq"
Jul 14 11:45:22.741111 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:22.741084    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jr8dp\" (UniqueName: \"kubernetes.io/projected/8e85fe8f-489a-470b-a22c-e01803a6709a-kube-api-access-jr8dp\") pod \"kuadrant-operator-controller-manager-8684c9ff49-726bq\" (UID: \"8e85fe8f-489a-470b-a22c-e01803a6709a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-726bq"
Jul 14 11:45:22.777111 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:22.777078    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:45:22.906116 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:22.906024    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-726bq"
Jul 14 11:45:23.012996 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:23.012960    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:45:23.044887 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:23.044857    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-726bq"]
Jul 14 11:45:23.048322 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:45:23.048293    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e85fe8f_489a_470b_a22c_e01803a6709a.slice/crio-5749eba8529adb1fabcb1523dcc0f7d215a93c841aadaa2f9b42d536d8cba482 WatchSource:0}: Error finding container 5749eba8529adb1fabcb1523dcc0f7d215a93c841aadaa2f9b42d536d8cba482: Status 404 returned error can't find the container with id 5749eba8529adb1fabcb1523dcc0f7d215a93c841aadaa2f9b42d536d8cba482
Jul 14 11:45:23.173051 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:23.172964    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:45:23.270507 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:23.270460    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:45:24.039388 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:24.039349    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-726bq" event={"ID":"8e85fe8f-489a-470b-a22c-e01803a6709a","Type":"ContainerStarted","Data":"5749eba8529adb1fabcb1523dcc0f7d215a93c841aadaa2f9b42d536d8cba482"}
Jul 14 11:45:25.846416 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:25.846377    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:45:27.427152 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:27.427058    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:45:27.561006 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:27.560970    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:45:28.011533 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:28.011502    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-m7jvb"
Jul 14 11:45:29.063698 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:29.063665    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-726bq" event={"ID":"8e85fe8f-489a-470b-a22c-e01803a6709a","Type":"ContainerStarted","Data":"e92b742628ef7dc812a4149d2cb67d2687ca1de2e1764f56bec4702342a3f35e"}
Jul 14 11:45:29.064078 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:29.063788    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-726bq"
Jul 14 11:45:29.097937 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:29.097885    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-726bq" podStartSLOduration=1.296589665 podStartE2EDuration="7.097867395s" totalImagesPullingTime="5.80127773s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 11:45:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 11:45:29.095962122 +0000 UTC m=+480.327771417" watchObservedRunningTime="2026-07-14 11:45:29.097867395 +0000 UTC m=+480.329676691"
Jul 14 11:45:31.076684 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:31.076643    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:45:31.479709 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:31.479665    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:45:31.659628 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:31.659596    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-9cpbg"]
Jul 14 11:45:31.662890 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:31.662873    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9cpbg"
Jul 14 11:45:31.665829 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:31.665802    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Jul 14 11:45:31.665829 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:31.665816    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-dbmqg\""
Jul 14 11:45:31.666032 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:31.665848    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Jul 14 11:45:31.680685 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:31.680656    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-9cpbg"]
Jul 14 11:45:31.699165 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:31.699127    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:45:31.701160 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:31.701139    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2ead8ac4-7d3e-4cff-8264-7e3229692eac-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-9cpbg\" (UID: \"2ead8ac4-7d3e-4cff-8264-7e3229692eac\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9cpbg"
Jul 14 11:45:31.701256 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:31.701193    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mh5jz\" (UniqueName: \"kubernetes.io/projected/2ead8ac4-7d3e-4cff-8264-7e3229692eac-kube-api-access-mh5jz\") pod \"kuadrant-console-plugin-66b5774985-9cpbg\" (UID: \"2ead8ac4-7d3e-4cff-8264-7e3229692eac\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9cpbg"
Jul 14 11:45:31.701321 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:31.701294    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/2ead8ac4-7d3e-4cff-8264-7e3229692eac-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-9cpbg\" (UID: \"2ead8ac4-7d3e-4cff-8264-7e3229692eac\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9cpbg"
Jul 14 11:45:31.802194 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:31.802097    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/2ead8ac4-7d3e-4cff-8264-7e3229692eac-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-9cpbg\" (UID: \"2ead8ac4-7d3e-4cff-8264-7e3229692eac\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9cpbg"
Jul 14 11:45:31.802194 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:31.802153    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2ead8ac4-7d3e-4cff-8264-7e3229692eac-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-9cpbg\" (UID: \"2ead8ac4-7d3e-4cff-8264-7e3229692eac\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9cpbg"
Jul 14 11:45:31.802393 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:31.802212    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mh5jz\" (UniqueName: \"kubernetes.io/projected/2ead8ac4-7d3e-4cff-8264-7e3229692eac-kube-api-access-mh5jz\") pod \"kuadrant-console-plugin-66b5774985-9cpbg\" (UID: \"2ead8ac4-7d3e-4cff-8264-7e3229692eac\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9cpbg"
Jul 14 11:45:31.802393 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:45:31.802246    2551 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Jul 14 11:45:31.802393 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:45:31.802322    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ead8ac4-7d3e-4cff-8264-7e3229692eac-plugin-serving-cert podName:2ead8ac4-7d3e-4cff-8264-7e3229692eac nodeName:}" failed. No retries permitted until 2026-07-14 11:45:32.302300724 +0000 UTC m=+483.534110005 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/2ead8ac4-7d3e-4cff-8264-7e3229692eac-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-9cpbg" (UID: "2ead8ac4-7d3e-4cff-8264-7e3229692eac") : secret "plugin-serving-cert" not found
Jul 14 11:45:31.802756 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:31.802738    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2ead8ac4-7d3e-4cff-8264-7e3229692eac-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-9cpbg\" (UID: \"2ead8ac4-7d3e-4cff-8264-7e3229692eac\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9cpbg"
Jul 14 11:45:31.814451 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:31.814420    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mh5jz\" (UniqueName: \"kubernetes.io/projected/2ead8ac4-7d3e-4cff-8264-7e3229692eac-kube-api-access-mh5jz\") pod \"kuadrant-console-plugin-66b5774985-9cpbg\" (UID: \"2ead8ac4-7d3e-4cff-8264-7e3229692eac\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9cpbg"
Jul 14 11:45:32.031422 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:32.031390    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rdgp4"
Jul 14 11:45:32.306305 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:32.306268    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/2ead8ac4-7d3e-4cff-8264-7e3229692eac-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-9cpbg\" (UID: \"2ead8ac4-7d3e-4cff-8264-7e3229692eac\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9cpbg"
Jul 14 11:45:32.308766 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:32.308735    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/2ead8ac4-7d3e-4cff-8264-7e3229692eac-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-9cpbg\" (UID: \"2ead8ac4-7d3e-4cff-8264-7e3229692eac\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9cpbg"
Jul 14 11:45:32.324472 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:32.324443    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:45:32.571696 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:32.571604    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9cpbg"
Jul 14 11:45:32.728137 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:32.728112    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-9cpbg"]
Jul 14 11:45:32.730544 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:45:32.730519    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2ead8ac4_7d3e_4cff_8264_7e3229692eac.slice/crio-ba324d3d52eaebef4a4520f8b14963b4cac6ec673867502449a57899067034cf WatchSource:0}: Error finding container ba324d3d52eaebef4a4520f8b14963b4cac6ec673867502449a57899067034cf: Status 404 returned error can't find the container with id ba324d3d52eaebef4a4520f8b14963b4cac6ec673867502449a57899067034cf
Jul 14 11:45:33.082329 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:33.082290    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9cpbg" event={"ID":"2ead8ac4-7d3e-4cff-8264-7e3229692eac","Type":"ContainerStarted","Data":"ba324d3d52eaebef4a4520f8b14963b4cac6ec673867502449a57899067034cf"}
Jul 14 11:45:34.436858 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:34.436817    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:45:34.554667 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:34.554624    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:45:40.070747 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:40.070716    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-726bq"
Jul 14 11:45:41.721568 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:41.721521    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-47skt"]
Jul 14 11:45:41.728352 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:41.728327    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-47skt"
Jul 14 11:45:41.740112 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:41.740082    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-47skt"]
Jul 14 11:45:41.787641 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:41.787573    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5fxc\" (UniqueName: \"kubernetes.io/projected/5d50e04a-f37d-4ef2-bc83-90091a1cb0fb-kube-api-access-f5fxc\") pod \"kuadrant-operator-controller-manager-55467d9f7-47skt\" (UID: \"5d50e04a-f37d-4ef2-bc83-90091a1cb0fb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-47skt"
Jul 14 11:45:41.787853 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:41.787733    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5d50e04a-f37d-4ef2-bc83-90091a1cb0fb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-47skt\" (UID: \"5d50e04a-f37d-4ef2-bc83-90091a1cb0fb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-47skt"
Jul 14 11:45:41.831109 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:41.831073    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-47skt"]
Jul 14 11:45:41.831385 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:45:41.831361    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[extensions-socket-volume kube-api-access-f5fxc], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-47skt" podUID="5d50e04a-f37d-4ef2-bc83-90091a1cb0fb"
Jul 14 11:45:41.876787 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:41.876747    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9df67"]
Jul 14 11:45:41.881721 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:41.881690    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9df67"
Jul 14 11:45:41.888458 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:41.888426    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5fxc\" (UniqueName: \"kubernetes.io/projected/5d50e04a-f37d-4ef2-bc83-90091a1cb0fb-kube-api-access-f5fxc\") pod \"kuadrant-operator-controller-manager-55467d9f7-47skt\" (UID: \"5d50e04a-f37d-4ef2-bc83-90091a1cb0fb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-47skt"
Jul 14 11:45:41.888636 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:41.888554    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5d50e04a-f37d-4ef2-bc83-90091a1cb0fb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-47skt\" (UID: \"5d50e04a-f37d-4ef2-bc83-90091a1cb0fb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-47skt"
Jul 14 11:45:41.889050 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:41.888984    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5d50e04a-f37d-4ef2-bc83-90091a1cb0fb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-47skt\" (UID: \"5d50e04a-f37d-4ef2-bc83-90091a1cb0fb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-47skt"
Jul 14 11:45:41.894053 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:41.894028    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9df67"]
Jul 14 11:45:41.904069 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:41.904035    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5fxc\" (UniqueName: \"kubernetes.io/projected/5d50e04a-f37d-4ef2-bc83-90091a1cb0fb-kube-api-access-f5fxc\") pod \"kuadrant-operator-controller-manager-55467d9f7-47skt\" (UID: \"5d50e04a-f37d-4ef2-bc83-90091a1cb0fb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-47skt"
Jul 14 11:45:41.914345 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:41.914307    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-47skt"]
Jul 14 11:45:41.930268 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:41.930233    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-47skt"]
Jul 14 11:45:41.943541 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:41.943498    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-726bq"]
Jul 14 11:45:41.943810 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:41.943757    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-726bq" podUID="8e85fe8f-489a-470b-a22c-e01803a6709a" containerName="manager" containerID="cri-o://e92b742628ef7dc812a4149d2cb67d2687ca1de2e1764f56bec4702342a3f35e" gracePeriod=2
Jul 14 11:45:41.950128 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:41.950093    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-726bq"]
Jul 14 11:45:41.963602 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:41.963466    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9df67"]
Jul 14 11:45:41.964013 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:45:41.963963    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[extensions-socket-volume kube-api-access-ffstt], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9df67" podUID="ed27bf6a-4f65-489d-a356-ffb891bdae79"
Jul 14 11:45:41.970346 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:41.970306    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7vxv7"]
Jul 14 11:45:41.970903 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:41.970752    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e85fe8f-489a-470b-a22c-e01803a6709a" containerName="manager"
Jul 14 11:45:41.970903 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:41.970771    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e85fe8f-489a-470b-a22c-e01803a6709a" containerName="manager"
Jul 14 11:45:41.970903 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:41.970871    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e85fe8f-489a-470b-a22c-e01803a6709a" containerName="manager"
Jul 14 11:45:41.973877 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:41.973807    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9df67"]
Jul 14 11:45:41.973973 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:41.973950    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7vxv7"
Jul 14 11:45:41.976862 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:45:41.976817    2551 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-726bq\" is forbidden: User \"system:node:ip-10-0-136-173.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-173.ec2.internal' and this object" podUID="8e85fe8f-489a-470b-a22c-e01803a6709a" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-726bq"
Jul 14 11:45:41.981628 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:41.981593    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7vxv7"]
Jul 14 11:45:41.985199 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:41.985175    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rdgp4"]
Jul 14 11:45:41.985438 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:41.985397    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rdgp4" podUID="f892a432-e9d2-451b-9a42-fdca3d188149" containerName="manager" containerID="cri-o://e2b9d9aa638d48077a32329966ca34b9bb5f1f928611c01dd3af971c7cd0f302" gracePeriod=2
Jul 14 11:45:41.989553 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:41.989499    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ed27bf6a-4f65-489d-a356-ffb891bdae79-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-9df67\" (UID: \"ed27bf6a-4f65-489d-a356-ffb891bdae79\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9df67"
Jul 14 11:45:41.989680 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:41.989619    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffstt\" (UniqueName: \"kubernetes.io/projected/ed27bf6a-4f65-489d-a356-ffb891bdae79-kube-api-access-ffstt\") pod \"kuadrant-operator-controller-manager-86878c769f-9df67\" (UID: \"ed27bf6a-4f65-489d-a356-ffb891bdae79\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9df67"
Jul 14 11:45:41.992616 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:41.992327    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-j7b55"]
Jul 14 11:45:41.995949 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:41.995919    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rdgp4"]
Jul 14 11:45:41.996061 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:41.996031    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-j7b55"
Jul 14 11:45:42.012804 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:42.012680    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-j7b55"]
Jul 14 11:45:42.014372 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:42.014347    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cqrmp"]
Jul 14 11:45:42.014935 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:42.014914    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f892a432-e9d2-451b-9a42-fdca3d188149" containerName="manager"
Jul 14 11:45:42.015059 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:42.014938    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="f892a432-e9d2-451b-9a42-fdca3d188149" containerName="manager"
Jul 14 11:45:42.015120 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:42.015088    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="f892a432-e9d2-451b-9a42-fdca3d188149" containerName="manager"
Jul 14 11:45:42.018242 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:42.018227    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cqrmp"
Jul 14 11:45:42.030213 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:45:42.030168    2551 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-726bq\" is forbidden: User \"system:node:ip-10-0-136-173.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-173.ec2.internal' and this object" podUID="8e85fe8f-489a-470b-a22c-e01803a6709a" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-726bq"
Jul 14 11:45:42.030330 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:42.030226    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rdgp4" podUID="f892a432-e9d2-451b-9a42-fdca3d188149" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.24:8081/readyz\": dial tcp 10.132.0.24:8081: connect: connection refused"
Jul 14 11:45:42.032518 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:45:42.032482    2551 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-rdgp4\" is forbidden: User \"system:node:ip-10-0-136-173.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-173.ec2.internal' and this object" podUID="f892a432-e9d2-451b-9a42-fdca3d188149" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rdgp4"
Jul 14 11:45:42.039550 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:42.039520    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cqrmp"]
Jul 14 11:45:42.056466 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:45:42.056425    2551 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-726bq\" is forbidden: User \"system:node:ip-10-0-136-173.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-173.ec2.internal' and this object" podUID="8e85fe8f-489a-470b-a22c-e01803a6709a" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-726bq"
Jul 14 11:45:42.058546 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:45:42.058505    2551 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-rdgp4\" is forbidden: User \"system:node:ip-10-0-136-173.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-173.ec2.internal' and this object" podUID="f892a432-e9d2-451b-9a42-fdca3d188149" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rdgp4"
Jul 14 11:45:42.090239 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:42.090207    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmgl2\" (UniqueName: \"kubernetes.io/projected/34006583-7b96-406e-81bc-8af75a8a7d46-kube-api-access-tmgl2\") pod \"kuadrant-operator-controller-manager-8684c9ff49-7vxv7\" (UID: \"34006583-7b96-406e-81bc-8af75a8a7d46\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7vxv7"
Jul 14 11:45:42.090367 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:42.090261    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5qmh\" (UniqueName: \"kubernetes.io/projected/4c16658c-2c58-4e12-9bca-cbb43c9e970e-kube-api-access-v5qmh\") pod \"kuadrant-operator-controller-manager-86878c769f-j7b55\" (UID: \"4c16658c-2c58-4e12-9bca-cbb43c9e970e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-j7b55"
Jul 14 11:45:42.090367 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:42.090298    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ed27bf6a-4f65-489d-a356-ffb891bdae79-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-9df67\" (UID: \"ed27bf6a-4f65-489d-a356-ffb891bdae79\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9df67"
Jul 14 11:45:42.090367 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:42.090327    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4c16658c-2c58-4e12-9bca-cbb43c9e970e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-j7b55\" (UID: \"4c16658c-2c58-4e12-9bca-cbb43c9e970e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-j7b55"
Jul 14 11:45:42.090367 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:42.090350    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffstt\" (UniqueName: \"kubernetes.io/projected/ed27bf6a-4f65-489d-a356-ffb891bdae79-kube-api-access-ffstt\") pod \"kuadrant-operator-controller-manager-86878c769f-9df67\" (UID: \"ed27bf6a-4f65-489d-a356-ffb891bdae79\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9df67"
Jul 14 11:45:42.090611 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:42.090375    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/34006583-7b96-406e-81bc-8af75a8a7d46-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-7vxv7\" (UID: \"34006583-7b96-406e-81bc-8af75a8a7d46\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7vxv7"
Jul 14 11:45:42.090611 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:42.090413    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rm5t\" (UniqueName: \"kubernetes.io/projected/3f3d650b-412d-4402-853e-e3b9b752a0ac-kube-api-access-4rm5t\") pod \"limitador-operator-controller-manager-859d7b78bd-cqrmp\" (UID: \"3f3d650b-412d-4402-853e-e3b9b752a0ac\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cqrmp"
Jul 14 11:45:42.090819 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:42.090786    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ed27bf6a-4f65-489d-a356-ffb891bdae79-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-9df67\" (UID: \"ed27bf6a-4f65-489d-a356-ffb891bdae79\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9df67"
Jul 14 11:45:42.095661 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:45:42.095640    2551 projected.go:196] Error preparing data for projected volume kube-api-access-ffstt for pod kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9df67: failed to fetch token: pod "kuadrant-operator-controller-manager-86878c769f-9df67" not found
Jul 14 11:45:42.095771 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:45:42.095747    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ed27bf6a-4f65-489d-a356-ffb891bdae79-kube-api-access-ffstt podName:ed27bf6a-4f65-489d-a356-ffb891bdae79 nodeName:}" failed. No retries permitted until 2026-07-14 11:45:42.595723552 +0000 UTC m=+493.827532829 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-ffstt" (UniqueName: "kubernetes.io/projected/ed27bf6a-4f65-489d-a356-ffb891bdae79-kube-api-access-ffstt") pod "kuadrant-operator-controller-manager-86878c769f-9df67" (UID: "ed27bf6a-4f65-489d-a356-ffb891bdae79") : failed to fetch token: pod "kuadrant-operator-controller-manager-86878c769f-9df67" not found
Jul 14 11:45:42.130011 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:42.129978    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9df67"
Jul 14 11:45:42.130187 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:42.129989    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-47skt"
Jul 14 11:45:42.132723 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:45:42.132690    2551 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-726bq\" is forbidden: User \"system:node:ip-10-0-136-173.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-173.ec2.internal' and this object" podUID="8e85fe8f-489a-470b-a22c-e01803a6709a" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-726bq"
Jul 14 11:45:42.134720 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:45:42.134683    2551 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-47skt\" is forbidden: User \"system:node:ip-10-0-136-173.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-173.ec2.internal' and this object" podUID="5d50e04a-f37d-4ef2-bc83-90091a1cb0fb" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-47skt"
Jul 14 11:45:42.136636 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:45:42.136603    2551 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-rdgp4\" is forbidden: User \"system:node:ip-10-0-136-173.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-173.ec2.internal' and this object" podUID="f892a432-e9d2-451b-9a42-fdca3d188149" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rdgp4"
Jul 14 11:45:42.138281 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:42.138262    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9df67"
Jul 14 11:45:42.138533 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:45:42.138495    2551 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-86878c769f-9df67\" is forbidden: User \"system:node:ip-10-0-136-173.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-173.ec2.internal' and this object" podUID="ed27bf6a-4f65-489d-a356-ffb891bdae79" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9df67"
Jul 14 11:45:42.140425 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:45:42.140400    2551 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-726bq\" is forbidden: User \"system:node:ip-10-0-136-173.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-173.ec2.internal' and this object" podUID="8e85fe8f-489a-470b-a22c-e01803a6709a" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-726bq"
Jul 14 11:45:42.142435 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:45:42.142404    2551 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-47skt\" is forbidden: User \"system:node:ip-10-0-136-173.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-173.ec2.internal' and this object" podUID="5d50e04a-f37d-4ef2-bc83-90091a1cb0fb" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-47skt"
Jul 14 11:45:42.142536 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:42.142471    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-47skt"
Jul 14 11:45:42.144507 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:45:42.144481    2551 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-rdgp4\" is forbidden: User \"system:node:ip-10-0-136-173.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-173.ec2.internal' and this object" podUID="f892a432-e9d2-451b-9a42-fdca3d188149" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rdgp4"
Jul 14 11:45:42.146550 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:45:42.146523    2551 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-86878c769f-9df67\" is forbidden: User \"system:node:ip-10-0-136-173.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-173.ec2.internal' and this object" podUID="ed27bf6a-4f65-489d-a356-ffb891bdae79" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9df67"
Jul 14 11:45:42.148606 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:45:42.148555    2551 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-726bq\" is forbidden: User \"system:node:ip-10-0-136-173.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-173.ec2.internal' and this object" podUID="8e85fe8f-489a-470b-a22c-e01803a6709a" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-726bq"
Jul 14 11:45:42.150750 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:45:42.150691    2551 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-47skt\" is forbidden: User \"system:node:ip-10-0-136-173.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-173.ec2.internal' and this object" podUID="5d50e04a-f37d-4ef2-bc83-90091a1cb0fb" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-47skt"
Jul 14 11:45:42.152712 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:45:42.152687    2551 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-rdgp4\" is forbidden: User \"system:node:ip-10-0-136-173.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-173.ec2.internal' and this object" podUID="f892a432-e9d2-451b-9a42-fdca3d188149" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rdgp4"
Jul 14 11:45:42.154757 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:45:42.154731    2551 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-86878c769f-9df67\" is forbidden: User \"system:node:ip-10-0-136-173.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-173.ec2.internal' and this object" podUID="ed27bf6a-4f65-489d-a356-ffb891bdae79" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9df67"
Jul 14 11:45:42.191288 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:42.191250    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ed27bf6a-4f65-489d-a356-ffb891bdae79-extensions-socket-volume\") pod \"ed27bf6a-4f65-489d-a356-ffb891bdae79\" (UID: \"ed27bf6a-4f65-489d-a356-ffb891bdae79\") "
Jul 14 11:45:42.191475 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:42.191390    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmgl2\" (UniqueName: \"kubernetes.io/projected/34006583-7b96-406e-81bc-8af75a8a7d46-kube-api-access-tmgl2\") pod \"kuadrant-operator-controller-manager-8684c9ff49-7vxv7\" (UID: \"34006583-7b96-406e-81bc-8af75a8a7d46\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7vxv7"
Jul 14 11:45:42.191475 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:42.191431    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5qmh\" (UniqueName: \"kubernetes.io/projected/4c16658c-2c58-4e12-9bca-cbb43c9e970e-kube-api-access-v5qmh\") pod \"kuadrant-operator-controller-manager-86878c769f-j7b55\" (UID: \"4c16658c-2c58-4e12-9bca-cbb43c9e970e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-j7b55"
Jul 14 11:45:42.191615 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:42.191478    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4c16658c-2c58-4e12-9bca-cbb43c9e970e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-j7b55\" (UID: \"4c16658c-2c58-4e12-9bca-cbb43c9e970e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-j7b55"
Jul 14 11:45:42.191615 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:42.191515    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/34006583-7b96-406e-81bc-8af75a8a7d46-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-7vxv7\" (UID: \"34006583-7b96-406e-81bc-8af75a8a7d46\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7vxv7"
Jul 14 11:45:42.191615 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:42.191558    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rm5t\" (UniqueName: \"kubernetes.io/projected/3f3d650b-412d-4402-853e-e3b9b752a0ac-kube-api-access-4rm5t\") pod \"limitador-operator-controller-manager-859d7b78bd-cqrmp\" (UID: \"3f3d650b-412d-4402-853e-e3b9b752a0ac\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cqrmp"
Jul 14 11:45:42.191615 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:42.191573    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed27bf6a-4f65-489d-a356-ffb891bdae79-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "ed27bf6a-4f65-489d-a356-ffb891bdae79" (UID: "ed27bf6a-4f65-489d-a356-ffb891bdae79"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 11:45:42.191803 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:42.191665    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffstt\" (UniqueName: \"kubernetes.io/projected/ed27bf6a-4f65-489d-a356-ffb891bdae79-kube-api-access-ffstt\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 11:45:42.191959 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:42.191912    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4c16658c-2c58-4e12-9bca-cbb43c9e970e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-j7b55\" (UID: \"4c16658c-2c58-4e12-9bca-cbb43c9e970e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-j7b55"
Jul 14 11:45:42.192095 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:42.192071    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/34006583-7b96-406e-81bc-8af75a8a7d46-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-7vxv7\" (UID: \"34006583-7b96-406e-81bc-8af75a8a7d46\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7vxv7"
Jul 14 11:45:42.208548 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:42.208515    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmgl2\" (UniqueName: \"kubernetes.io/projected/34006583-7b96-406e-81bc-8af75a8a7d46-kube-api-access-tmgl2\") pod \"kuadrant-operator-controller-manager-8684c9ff49-7vxv7\" (UID: \"34006583-7b96-406e-81bc-8af75a8a7d46\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7vxv7"
Jul 14 11:45:42.209433 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:42.209394    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5qmh\" (UniqueName: \"kubernetes.io/projected/4c16658c-2c58-4e12-9bca-cbb43c9e970e-kube-api-access-v5qmh\") pod \"kuadrant-operator-controller-manager-86878c769f-j7b55\" (UID: \"4c16658c-2c58-4e12-9bca-cbb43c9e970e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-j7b55"
Jul 14 11:45:42.209550 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:42.209449    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rm5t\" (UniqueName: \"kubernetes.io/projected/3f3d650b-412d-4402-853e-e3b9b752a0ac-kube-api-access-4rm5t\") pod \"limitador-operator-controller-manager-859d7b78bd-cqrmp\" (UID: \"3f3d650b-412d-4402-853e-e3b9b752a0ac\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cqrmp"
Jul 14 11:45:42.292509 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:42.292429    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5d50e04a-f37d-4ef2-bc83-90091a1cb0fb-extensions-socket-volume\") pod \"5d50e04a-f37d-4ef2-bc83-90091a1cb0fb\" (UID: \"5d50e04a-f37d-4ef2-bc83-90091a1cb0fb\") "
Jul 14 11:45:42.292509 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:42.292484    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f5fxc\" (UniqueName: \"kubernetes.io/projected/5d50e04a-f37d-4ef2-bc83-90091a1cb0fb-kube-api-access-f5fxc\") pod \"5d50e04a-f37d-4ef2-bc83-90091a1cb0fb\" (UID: \"5d50e04a-f37d-4ef2-bc83-90091a1cb0fb\") "
Jul 14 11:45:42.292782 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:42.292749    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d50e04a-f37d-4ef2-bc83-90091a1cb0fb-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "5d50e04a-f37d-4ef2-bc83-90091a1cb0fb" (UID: "5d50e04a-f37d-4ef2-bc83-90091a1cb0fb"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 11:45:42.292847 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:42.292784    2551 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ed27bf6a-4f65-489d-a356-ffb891bdae79-extensions-socket-volume\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 11:45:42.294957 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:42.294932    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d50e04a-f37d-4ef2-bc83-90091a1cb0fb-kube-api-access-f5fxc" (OuterVolumeSpecName: "kube-api-access-f5fxc") pod "5d50e04a-f37d-4ef2-bc83-90091a1cb0fb" (UID: "5d50e04a-f37d-4ef2-bc83-90091a1cb0fb"). InnerVolumeSpecName "kube-api-access-f5fxc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 11:45:42.375920 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:42.375887    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7vxv7"
Jul 14 11:45:42.385556 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:42.385526    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-j7b55"
Jul 14 11:45:42.393549 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:42.393520    2551 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5d50e04a-f37d-4ef2-bc83-90091a1cb0fb-extensions-socket-volume\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 11:45:42.393549 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:42.393550    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5fxc\" (UniqueName: \"kubernetes.io/projected/5d50e04a-f37d-4ef2-bc83-90091a1cb0fb-kube-api-access-f5fxc\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 11:45:42.394409 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:42.394380    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cqrmp"
Jul 14 11:45:42.953256 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:42.953197    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:45:43.060666 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:43.060618    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:45:43.140814 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:43.140785    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-47skt"
Jul 14 11:45:43.140814 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:43.140799    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9df67"
Jul 14 11:45:43.143634 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:45:43.143589    2551 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-726bq\" is forbidden: User \"system:node:ip-10-0-136-173.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-173.ec2.internal' and this object" podUID="8e85fe8f-489a-470b-a22c-e01803a6709a" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-726bq"
Jul 14 11:45:43.145742 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:45:43.145707    2551 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-47skt\" is forbidden: User \"system:node:ip-10-0-136-173.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-173.ec2.internal' and this object" podUID="5d50e04a-f37d-4ef2-bc83-90091a1cb0fb" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-47skt"
Jul 14 11:45:43.147863 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:45:43.147839    2551 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-rdgp4\" is forbidden: User \"system:node:ip-10-0-136-173.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-173.ec2.internal' and this object" podUID="f892a432-e9d2-451b-9a42-fdca3d188149" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rdgp4"
Jul 14 11:45:43.150116 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:45:43.150082    2551 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-86878c769f-9df67\" is forbidden: User \"system:node:ip-10-0-136-173.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-173.ec2.internal' and this object" podUID="ed27bf6a-4f65-489d-a356-ffb891bdae79" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9df67"
Jul 14 11:45:43.152287 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:45:43.152258    2551 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-726bq\" is forbidden: User \"system:node:ip-10-0-136-173.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-173.ec2.internal' and this object" podUID="8e85fe8f-489a-470b-a22c-e01803a6709a" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-726bq"
Jul 14 11:45:43.154333 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:45:43.154303    2551 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-47skt\" is forbidden: User \"system:node:ip-10-0-136-173.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-173.ec2.internal' and this object" podUID="5d50e04a-f37d-4ef2-bc83-90091a1cb0fb" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-47skt"
Jul 14 11:45:43.156477 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:45:43.156447    2551 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-rdgp4\" is forbidden: User \"system:node:ip-10-0-136-173.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-173.ec2.internal' and this object" podUID="f892a432-e9d2-451b-9a42-fdca3d188149" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rdgp4"
Jul 14 11:45:43.158737 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:45:43.158701    2551 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-86878c769f-9df67\" is forbidden: User \"system:node:ip-10-0-136-173.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-173.ec2.internal' and this object" podUID="ed27bf6a-4f65-489d-a356-ffb891bdae79" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9df67"
Jul 14 11:45:43.160898 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:45:43.160875    2551 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-726bq\" is forbidden: User \"system:node:ip-10-0-136-173.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-173.ec2.internal' and this object" podUID="8e85fe8f-489a-470b-a22c-e01803a6709a" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-726bq"
Jul 14 11:45:43.163326 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:45:43.163300    2551 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-47skt\" is forbidden: User \"system:node:ip-10-0-136-173.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-173.ec2.internal' and this object" podUID="5d50e04a-f37d-4ef2-bc83-90091a1cb0fb" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-47skt"
Jul 14 11:45:43.165557 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:45:43.165536    2551 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-rdgp4\" is forbidden: User \"system:node:ip-10-0-136-173.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-173.ec2.internal' and this object" podUID="f892a432-e9d2-451b-9a42-fdca3d188149" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rdgp4"
Jul 14 11:45:43.168334 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:45:43.168310    2551 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-86878c769f-9df67\" is forbidden: User \"system:node:ip-10-0-136-173.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-173.ec2.internal' and this object" podUID="ed27bf6a-4f65-489d-a356-ffb891bdae79" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9df67"
Jul 14 11:45:43.192680 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:43.192649    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d50e04a-f37d-4ef2-bc83-90091a1cb0fb" path="/var/lib/kubelet/pods/5d50e04a-f37d-4ef2-bc83-90091a1cb0fb/volumes"
Jul 14 11:45:43.192976 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:43.192958    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed27bf6a-4f65-489d-a356-ffb891bdae79" path="/var/lib/kubelet/pods/ed27bf6a-4f65-489d-a356-ffb891bdae79/volumes"
Jul 14 11:45:45.582986 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:45.582945    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:45:45.682314 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:45.682280    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:45:48.787681 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:48.787642    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:45:48.943355 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:48.943319    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:45:51.342783 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:51.342744    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:45:52.553926 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:52.553878    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:45:52.649552 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:52.649508    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:45:56.492806 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:56.492762    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cqrmp"]
Jul 14 11:45:56.496338 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:45:56.496300    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f3d650b_412d_4402_853e_e3b9b752a0ac.slice/crio-d32d5f6393ca526b250d2105c9c1b17c2c32d34d82dfaf3fdb78465b48a680ca WatchSource:0}: Error finding container d32d5f6393ca526b250d2105c9c1b17c2c32d34d82dfaf3fdb78465b48a680ca: Status 404 returned error can't find the container with id d32d5f6393ca526b250d2105c9c1b17c2c32d34d82dfaf3fdb78465b48a680ca
Jul 14 11:45:56.514045 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:56.514014    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7vxv7"]
Jul 14 11:45:56.514911 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:56.514885    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-726bq"
Jul 14 11:45:56.517280 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:45:56.517249    2551 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-726bq\" is forbidden: User \"system:node:ip-10-0-136-173.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-173.ec2.internal' and this object" podUID="8e85fe8f-489a-470b-a22c-e01803a6709a" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-726bq"
Jul 14 11:45:56.523662 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:45:56.523633    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34006583_7b96_406e_81bc_8af75a8a7d46.slice/crio-64fef2456bffff8560b24261981b8c2b63ebdfe00b248ba9339b7fcafc962823 WatchSource:0}: Error finding container 64fef2456bffff8560b24261981b8c2b63ebdfe00b248ba9339b7fcafc962823: Status 404 returned error can't find the container with id 64fef2456bffff8560b24261981b8c2b63ebdfe00b248ba9339b7fcafc962823
Jul 14 11:45:56.538787 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:56.538758    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-j7b55"]
Jul 14 11:45:56.539198 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:56.539159    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rdgp4"
Jul 14 11:45:56.541247 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:45:56.541216    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c16658c_2c58_4e12_9bca_cbb43c9e970e.slice/crio-b0640ed931658218695b2c5f65aeffeb230c3566a9478df3a52b1c2acbb285ca WatchSource:0}: Error finding container b0640ed931658218695b2c5f65aeffeb230c3566a9478df3a52b1c2acbb285ca: Status 404 returned error can't find the container with id b0640ed931658218695b2c5f65aeffeb230c3566a9478df3a52b1c2acbb285ca
Jul 14 11:45:56.541480 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:45:56.541451    2551 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-rdgp4\" is forbidden: User \"system:node:ip-10-0-136-173.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-173.ec2.internal' and this object" podUID="f892a432-e9d2-451b-9a42-fdca3d188149" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rdgp4"
Jul 14 11:45:56.543704 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:45:56.543661    2551 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-726bq\" is forbidden: User \"system:node:ip-10-0-136-173.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-173.ec2.internal' and this object" podUID="8e85fe8f-489a-470b-a22c-e01803a6709a" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-726bq"
Jul 14 11:45:56.622591 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:56.622552    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8e85fe8f-489a-470b-a22c-e01803a6709a-extensions-socket-volume\") pod \"8e85fe8f-489a-470b-a22c-e01803a6709a\" (UID: \"8e85fe8f-489a-470b-a22c-e01803a6709a\") "
Jul 14 11:45:56.622713 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:56.622614    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jr8dp\" (UniqueName: \"kubernetes.io/projected/8e85fe8f-489a-470b-a22c-e01803a6709a-kube-api-access-jr8dp\") pod \"8e85fe8f-489a-470b-a22c-e01803a6709a\" (UID: \"8e85fe8f-489a-470b-a22c-e01803a6709a\") "
Jul 14 11:45:56.623126 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:56.623098    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e85fe8f-489a-470b-a22c-e01803a6709a-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "8e85fe8f-489a-470b-a22c-e01803a6709a" (UID: "8e85fe8f-489a-470b-a22c-e01803a6709a"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 11:45:56.625044 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:56.625019    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e85fe8f-489a-470b-a22c-e01803a6709a-kube-api-access-jr8dp" (OuterVolumeSpecName: "kube-api-access-jr8dp") pod "8e85fe8f-489a-470b-a22c-e01803a6709a" (UID: "8e85fe8f-489a-470b-a22c-e01803a6709a"). InnerVolumeSpecName "kube-api-access-jr8dp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 11:45:56.723485 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:56.723454    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-57ngl\" (UniqueName: \"kubernetes.io/projected/f892a432-e9d2-451b-9a42-fdca3d188149-kube-api-access-57ngl\") pod \"f892a432-e9d2-451b-9a42-fdca3d188149\" (UID: \"f892a432-e9d2-451b-9a42-fdca3d188149\") "
Jul 14 11:45:56.723671 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:56.723661    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jr8dp\" (UniqueName: \"kubernetes.io/projected/8e85fe8f-489a-470b-a22c-e01803a6709a-kube-api-access-jr8dp\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 11:45:56.723732 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:56.723679    2551 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8e85fe8f-489a-470b-a22c-e01803a6709a-extensions-socket-volume\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 11:45:56.725789 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:56.725757    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f892a432-e9d2-451b-9a42-fdca3d188149-kube-api-access-57ngl" (OuterVolumeSpecName: "kube-api-access-57ngl") pod "f892a432-e9d2-451b-9a42-fdca3d188149" (UID: "f892a432-e9d2-451b-9a42-fdca3d188149"). InnerVolumeSpecName "kube-api-access-57ngl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 11:45:56.824219 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:56.824117    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57ngl\" (UniqueName: \"kubernetes.io/projected/f892a432-e9d2-451b-9a42-fdca3d188149-kube-api-access-57ngl\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 11:45:57.190765 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:57.190732    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e85fe8f-489a-470b-a22c-e01803a6709a" path="/var/lib/kubelet/pods/8e85fe8f-489a-470b-a22c-e01803a6709a/volumes"
Jul 14 11:45:57.191073 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:57.191059    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f892a432-e9d2-451b-9a42-fdca3d188149" path="/var/lib/kubelet/pods/f892a432-e9d2-451b-9a42-fdca3d188149/volumes"
Jul 14 11:45:57.210288 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:57.210259    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cqrmp" event={"ID":"3f3d650b-412d-4402-853e-e3b9b752a0ac","Type":"ContainerStarted","Data":"1b4fb3bc1007493f89962959d0fb712a35222147058a6589133dbc0aaf0c6dee"}
Jul 14 11:45:57.210409 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:57.210309    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cqrmp" event={"ID":"3f3d650b-412d-4402-853e-e3b9b752a0ac","Type":"ContainerStarted","Data":"d32d5f6393ca526b250d2105c9c1b17c2c32d34d82dfaf3fdb78465b48a680ca"}
Jul 14 11:45:57.210409 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:57.210377    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cqrmp"
Jul 14 11:45:57.211686 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:57.211663    2551 generic.go:356] "Generic (PLEG): container finished" podID="8e85fe8f-489a-470b-a22c-e01803a6709a" containerID="e92b742628ef7dc812a4149d2cb67d2687ca1de2e1764f56bec4702342a3f35e" exitCode=0
Jul 14 11:45:57.211803 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:57.211698    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-726bq"
Jul 14 11:45:57.211803 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:57.211735    2551 scope.go:117] "RemoveContainer" containerID="e92b742628ef7dc812a4149d2cb67d2687ca1de2e1764f56bec4702342a3f35e"
Jul 14 11:45:57.213859 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:57.213473    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7vxv7" event={"ID":"34006583-7b96-406e-81bc-8af75a8a7d46","Type":"ContainerStarted","Data":"afb60b06846c8bcd3e324b6e639b9538f5a7b501b1a7ca6776e3dace632dd183"}
Jul 14 11:45:57.213859 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:57.213523    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7vxv7" event={"ID":"34006583-7b96-406e-81bc-8af75a8a7d46","Type":"ContainerStarted","Data":"64fef2456bffff8560b24261981b8c2b63ebdfe00b248ba9339b7fcafc962823"}
Jul 14 11:45:57.213859 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:57.213552    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7vxv7"
Jul 14 11:45:57.215405 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:57.215379    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-j7b55" event={"ID":"4c16658c-2c58-4e12-9bca-cbb43c9e970e","Type":"ContainerStarted","Data":"a8eb4a46c9046a40183b328e4451b682d22c48dd5e4666628d77a97e20b0eedf"}
Jul 14 11:45:57.215492 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:57.215419    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-j7b55" event={"ID":"4c16658c-2c58-4e12-9bca-cbb43c9e970e","Type":"ContainerStarted","Data":"b0640ed931658218695b2c5f65aeffeb230c3566a9478df3a52b1c2acbb285ca"}
Jul 14 11:45:57.215545 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:57.215488    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-j7b55"
Jul 14 11:45:57.217105 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:57.217082    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9cpbg" event={"ID":"2ead8ac4-7d3e-4cff-8264-7e3229692eac","Type":"ContainerStarted","Data":"4c669ef3a8d0e291d9dc32ccb2c7eea9ca04901bfa23b720bcb0d75c66357bbd"}
Jul 14 11:45:57.218352 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:57.218328    2551 generic.go:356] "Generic (PLEG): container finished" podID="f892a432-e9d2-451b-9a42-fdca3d188149" containerID="e2b9d9aa638d48077a32329966ca34b9bb5f1f928611c01dd3af971c7cd0f302" exitCode=0
Jul 14 11:45:57.218456 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:57.218359    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rdgp4"
Jul 14 11:45:57.221097 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:57.221083    2551 scope.go:117] "RemoveContainer" containerID="e92b742628ef7dc812a4149d2cb67d2687ca1de2e1764f56bec4702342a3f35e"
Jul 14 11:45:57.221320 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:45:57.221303    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e92b742628ef7dc812a4149d2cb67d2687ca1de2e1764f56bec4702342a3f35e\": container with ID starting with e92b742628ef7dc812a4149d2cb67d2687ca1de2e1764f56bec4702342a3f35e not found: ID does not exist" containerID="e92b742628ef7dc812a4149d2cb67d2687ca1de2e1764f56bec4702342a3f35e"
Jul 14 11:45:57.221360 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:57.221329    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e92b742628ef7dc812a4149d2cb67d2687ca1de2e1764f56bec4702342a3f35e"} err="failed to get container status \"e92b742628ef7dc812a4149d2cb67d2687ca1de2e1764f56bec4702342a3f35e\": rpc error: code = NotFound desc = could not find container \"e92b742628ef7dc812a4149d2cb67d2687ca1de2e1764f56bec4702342a3f35e\": container with ID starting with e92b742628ef7dc812a4149d2cb67d2687ca1de2e1764f56bec4702342a3f35e not found: ID does not exist"
Jul 14 11:45:57.221360 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:57.221347    2551 scope.go:117] "RemoveContainer" containerID="e2b9d9aa638d48077a32329966ca34b9bb5f1f928611c01dd3af971c7cd0f302"
Jul 14 11:45:57.232952 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:57.232934    2551 scope.go:117] "RemoveContainer" containerID="e2b9d9aa638d48077a32329966ca34b9bb5f1f928611c01dd3af971c7cd0f302"
Jul 14 11:45:57.233276 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:45:57.233250    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2b9d9aa638d48077a32329966ca34b9bb5f1f928611c01dd3af971c7cd0f302\": container with ID starting with e2b9d9aa638d48077a32329966ca34b9bb5f1f928611c01dd3af971c7cd0f302 not found: ID does not exist" containerID="e2b9d9aa638d48077a32329966ca34b9bb5f1f928611c01dd3af971c7cd0f302"
Jul 14 11:45:57.233363 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:57.233283    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2b9d9aa638d48077a32329966ca34b9bb5f1f928611c01dd3af971c7cd0f302"} err="failed to get container status \"e2b9d9aa638d48077a32329966ca34b9bb5f1f928611c01dd3af971c7cd0f302\": rpc error: code = NotFound desc = could not find container \"e2b9d9aa638d48077a32329966ca34b9bb5f1f928611c01dd3af971c7cd0f302\": container with ID starting with e2b9d9aa638d48077a32329966ca34b9bb5f1f928611c01dd3af971c7cd0f302 not found: ID does not exist"
Jul 14 11:45:57.248395 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:57.248337    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cqrmp" podStartSLOduration=16.248320454 podStartE2EDuration="16.248320454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 11:45:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 11:45:57.228856434 +0000 UTC m=+508.460665729" watchObservedRunningTime="2026-07-14 11:45:57.248320454 +0000 UTC m=+508.480129751"
Jul 14 11:45:57.248642 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:57.248443    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7vxv7" podStartSLOduration=16.248438762 podStartE2EDuration="16.248438762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 11:45:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 11:45:57.247202273 +0000 UTC m=+508.479011569" watchObservedRunningTime="2026-07-14 11:45:57.248438762 +0000 UTC m=+508.480248058"
Jul 14 11:45:57.267518 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:57.267463    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-j7b55" podStartSLOduration=16.267444155 podStartE2EDuration="16.267444155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 11:45:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 11:45:57.266835286 +0000 UTC m=+508.498644601" watchObservedRunningTime="2026-07-14 11:45:57.267444155 +0000 UTC m=+508.499253452"
Jul 14 11:45:57.268965 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:45:57.268932    2551 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-726bq\" is forbidden: User \"system:node:ip-10-0-136-173.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-173.ec2.internal' and this object" podUID="8e85fe8f-489a-470b-a22c-e01803a6709a" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-726bq"
Jul 14 11:45:57.289501 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:45:57.289372    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9cpbg" podStartSLOduration=2.598795485 podStartE2EDuration="26.289351961s" totalImagesPullingTime="23.690556476s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 11:45:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 11:45:57.288180494 +0000 UTC m=+508.519989791" watchObservedRunningTime="2026-07-14 11:45:57.289351961 +0000 UTC m=+508.521161260"
Jul 14 11:45:57.290295 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:45:57.290261    2551 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-rdgp4\" is forbidden: User \"system:node:ip-10-0-136-173.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-173.ec2.internal' and this object" podUID="f892a432-e9d2-451b-9a42-fdca3d188149" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rdgp4"
Jul 14 11:45:59.188969 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:45:59.188926    2551 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-726bq\" is forbidden: User \"system:node:ip-10-0-136-173.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-173.ec2.internal' and this object" podUID="8e85fe8f-489a-470b-a22c-e01803a6709a" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-726bq"
Jul 14 11:45:59.191144 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:45:59.191117    2551 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-rdgp4\" is forbidden: User \"system:node:ip-10-0-136-173.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-173.ec2.internal' and this object" podUID="f892a432-e9d2-451b-9a42-fdca3d188149" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rdgp4"
Jul 14 11:46:02.191327 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:02.191292    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:46:02.299268 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:02.299235    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:46:02.407503 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:02.407467    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:46:02.530186 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:02.530058    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:46:08.229671 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:08.229638    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cqrmp"
Jul 14 11:46:08.230095 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:08.229712    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-j7b55"
Jul 14 11:46:08.230095 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:08.229744    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7vxv7"
Jul 14 11:46:08.341306 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:08.341273    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7vxv7"]
Jul 14 11:46:08.341477 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:08.341455    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7vxv7" podUID="34006583-7b96-406e-81bc-8af75a8a7d46" containerName="manager" containerID="cri-o://afb60b06846c8bcd3e324b6e639b9538f5a7b501b1a7ca6776e3dace632dd183" gracePeriod=10
Jul 14 11:46:08.592507 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:08.592480    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7vxv7"
Jul 14 11:46:08.621542 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:08.621509    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/34006583-7b96-406e-81bc-8af75a8a7d46-extensions-socket-volume\") pod \"34006583-7b96-406e-81bc-8af75a8a7d46\" (UID: \"34006583-7b96-406e-81bc-8af75a8a7d46\") "
Jul 14 11:46:08.621715 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:08.621608    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tmgl2\" (UniqueName: \"kubernetes.io/projected/34006583-7b96-406e-81bc-8af75a8a7d46-kube-api-access-tmgl2\") pod \"34006583-7b96-406e-81bc-8af75a8a7d46\" (UID: \"34006583-7b96-406e-81bc-8af75a8a7d46\") "
Jul 14 11:46:08.621856 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:08.621834    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34006583-7b96-406e-81bc-8af75a8a7d46-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "34006583-7b96-406e-81bc-8af75a8a7d46" (UID: "34006583-7b96-406e-81bc-8af75a8a7d46"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 11:46:08.623629 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:08.623598    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34006583-7b96-406e-81bc-8af75a8a7d46-kube-api-access-tmgl2" (OuterVolumeSpecName: "kube-api-access-tmgl2") pod "34006583-7b96-406e-81bc-8af75a8a7d46" (UID: "34006583-7b96-406e-81bc-8af75a8a7d46"). InnerVolumeSpecName "kube-api-access-tmgl2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 11:46:08.722296 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:08.722259    2551 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/34006583-7b96-406e-81bc-8af75a8a7d46-extensions-socket-volume\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 11:46:08.722296 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:08.722289    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmgl2\" (UniqueName: \"kubernetes.io/projected/34006583-7b96-406e-81bc-8af75a8a7d46-kube-api-access-tmgl2\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 11:46:09.280440 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:09.280399    2551 generic.go:356] "Generic (PLEG): container finished" podID="34006583-7b96-406e-81bc-8af75a8a7d46" containerID="afb60b06846c8bcd3e324b6e639b9538f5a7b501b1a7ca6776e3dace632dd183" exitCode=0
Jul 14 11:46:09.280893 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:09.280449    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7vxv7"
Jul 14 11:46:09.280893 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:09.280472    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7vxv7" event={"ID":"34006583-7b96-406e-81bc-8af75a8a7d46","Type":"ContainerDied","Data":"afb60b06846c8bcd3e324b6e639b9538f5a7b501b1a7ca6776e3dace632dd183"}
Jul 14 11:46:09.280893 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:09.280526    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7vxv7" event={"ID":"34006583-7b96-406e-81bc-8af75a8a7d46","Type":"ContainerDied","Data":"64fef2456bffff8560b24261981b8c2b63ebdfe00b248ba9339b7fcafc962823"}
Jul 14 11:46:09.280893 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:09.280556    2551 scope.go:117] "RemoveContainer" containerID="afb60b06846c8bcd3e324b6e639b9538f5a7b501b1a7ca6776e3dace632dd183"
Jul 14 11:46:09.289040 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:09.289020    2551 scope.go:117] "RemoveContainer" containerID="afb60b06846c8bcd3e324b6e639b9538f5a7b501b1a7ca6776e3dace632dd183"
Jul 14 11:46:09.289313 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:46:09.289294    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afb60b06846c8bcd3e324b6e639b9538f5a7b501b1a7ca6776e3dace632dd183\": container with ID starting with afb60b06846c8bcd3e324b6e639b9538f5a7b501b1a7ca6776e3dace632dd183 not found: ID does not exist" containerID="afb60b06846c8bcd3e324b6e639b9538f5a7b501b1a7ca6776e3dace632dd183"
Jul 14 11:46:09.289356 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:09.289322    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afb60b06846c8bcd3e324b6e639b9538f5a7b501b1a7ca6776e3dace632dd183"} err="failed to get container status \"afb60b06846c8bcd3e324b6e639b9538f5a7b501b1a7ca6776e3dace632dd183\": rpc error: code = NotFound desc = could not find container \"afb60b06846c8bcd3e324b6e639b9538f5a7b501b1a7ca6776e3dace632dd183\": container with ID starting with afb60b06846c8bcd3e324b6e639b9538f5a7b501b1a7ca6776e3dace632dd183 not found: ID does not exist"
Jul 14 11:46:09.306772 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:09.306739    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7vxv7"]
Jul 14 11:46:09.313742 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:09.313719    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7vxv7"]
Jul 14 11:46:11.190742 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:11.190710    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34006583-7b96-406e-81bc-8af75a8a7d46" path="/var/lib/kubelet/pods/34006583-7b96-406e-81bc-8af75a8a7d46/volumes"
Jul 14 11:46:18.074194 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:18.074157    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:46:24.765442 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:24.765359    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-r7zl5"]
Jul 14 11:46:24.765806 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:24.765748    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34006583-7b96-406e-81bc-8af75a8a7d46" containerName="manager"
Jul 14 11:46:24.765806 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:24.765761    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="34006583-7b96-406e-81bc-8af75a8a7d46" containerName="manager"
Jul 14 11:46:24.765878 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:24.765821    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="34006583-7b96-406e-81bc-8af75a8a7d46" containerName="manager"
Jul 14 11:46:24.806307 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:24.806234    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:46:24.814473 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:24.814444    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-r7zl5"]
Jul 14 11:46:24.814665 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:24.814548    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-r7zl5"
Jul 14 11:46:24.817313 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:24.817290    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-2pc5j\""
Jul 14 11:46:24.817441 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:24.817292    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 14 11:46:24.852062 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:24.851994    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b49d636-77b9-487e-aa76-05a41fe47121-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-r7zl5\" (UID: \"2b49d636-77b9-487e-aa76-05a41fe47121\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-r7zl5"
Jul 14 11:46:24.852256 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:24.852091    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b49d636-77b9-487e-aa76-05a41fe47121-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-r7zl5\" (UID: \"2b49d636-77b9-487e-aa76-05a41fe47121\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-r7zl5"
Jul 14 11:46:24.852256 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:24.852152    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b49d636-77b9-487e-aa76-05a41fe47121-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-r7zl5\" (UID: \"2b49d636-77b9-487e-aa76-05a41fe47121\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-r7zl5"
Jul 14 11:46:24.852256 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:24.852204    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b49d636-77b9-487e-aa76-05a41fe47121-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-r7zl5\" (UID: \"2b49d636-77b9-487e-aa76-05a41fe47121\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-r7zl5"
Jul 14 11:46:24.852256 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:24.852243    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b49d636-77b9-487e-aa76-05a41fe47121-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-r7zl5\" (UID: \"2b49d636-77b9-487e-aa76-05a41fe47121\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-r7zl5"
Jul 14 11:46:24.852489 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:24.852271    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b49d636-77b9-487e-aa76-05a41fe47121-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-r7zl5\" (UID: \"2b49d636-77b9-487e-aa76-05a41fe47121\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-r7zl5"
Jul 14 11:46:24.852489 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:24.852305    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b49d636-77b9-487e-aa76-05a41fe47121-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-r7zl5\" (UID: \"2b49d636-77b9-487e-aa76-05a41fe47121\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-r7zl5"
Jul 14 11:46:24.852489 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:24.852409    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmjbl\" (UniqueName: \"kubernetes.io/projected/2b49d636-77b9-487e-aa76-05a41fe47121-kube-api-access-mmjbl\") pod \"maas-default-gateway-openshift-default-56654dcf7-r7zl5\" (UID: \"2b49d636-77b9-487e-aa76-05a41fe47121\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-r7zl5"
Jul 14 11:46:24.852489 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:24.852467    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b49d636-77b9-487e-aa76-05a41fe47121-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-r7zl5\" (UID: \"2b49d636-77b9-487e-aa76-05a41fe47121\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-r7zl5"
Jul 14 11:46:24.953823 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:24.953788    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b49d636-77b9-487e-aa76-05a41fe47121-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-r7zl5\" (UID: \"2b49d636-77b9-487e-aa76-05a41fe47121\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-r7zl5"
Jul 14 11:46:24.954006 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:24.953836    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b49d636-77b9-487e-aa76-05a41fe47121-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-r7zl5\" (UID: \"2b49d636-77b9-487e-aa76-05a41fe47121\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-r7zl5"
Jul 14 11:46:24.954006 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:24.953855    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b49d636-77b9-487e-aa76-05a41fe47121-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-r7zl5\" (UID: \"2b49d636-77b9-487e-aa76-05a41fe47121\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-r7zl5"
Jul 14 11:46:24.954006 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:24.953900    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b49d636-77b9-487e-aa76-05a41fe47121-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-r7zl5\" (UID: \"2b49d636-77b9-487e-aa76-05a41fe47121\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-r7zl5"
Jul 14 11:46:24.954006 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:24.953947    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b49d636-77b9-487e-aa76-05a41fe47121-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-r7zl5\" (UID: \"2b49d636-77b9-487e-aa76-05a41fe47121\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-r7zl5"
Jul 14 11:46:24.954006 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:24.953974    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b49d636-77b9-487e-aa76-05a41fe47121-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-r7zl5\" (UID: \"2b49d636-77b9-487e-aa76-05a41fe47121\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-r7zl5"
Jul 14 11:46:24.954006 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:24.953995    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b49d636-77b9-487e-aa76-05a41fe47121-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-r7zl5\" (UID: \"2b49d636-77b9-487e-aa76-05a41fe47121\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-r7zl5"
Jul 14 11:46:24.954295 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:24.954018    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b49d636-77b9-487e-aa76-05a41fe47121-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-r7zl5\" (UID: \"2b49d636-77b9-487e-aa76-05a41fe47121\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-r7zl5"
Jul 14 11:46:24.954295 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:24.954049    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmjbl\" (UniqueName: \"kubernetes.io/projected/2b49d636-77b9-487e-aa76-05a41fe47121-kube-api-access-mmjbl\") pod \"maas-default-gateway-openshift-default-56654dcf7-r7zl5\" (UID: \"2b49d636-77b9-487e-aa76-05a41fe47121\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-r7zl5"
Jul 14 11:46:24.954402 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:24.954298    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b49d636-77b9-487e-aa76-05a41fe47121-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-r7zl5\" (UID: \"2b49d636-77b9-487e-aa76-05a41fe47121\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-r7zl5"
Jul 14 11:46:24.954402 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:24.954348    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b49d636-77b9-487e-aa76-05a41fe47121-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-r7zl5\" (UID: \"2b49d636-77b9-487e-aa76-05a41fe47121\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-r7zl5"
Jul 14 11:46:24.954620 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:24.954595    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b49d636-77b9-487e-aa76-05a41fe47121-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-r7zl5\" (UID: \"2b49d636-77b9-487e-aa76-05a41fe47121\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-r7zl5"
Jul 14 11:46:24.954805 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:24.954574    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b49d636-77b9-487e-aa76-05a41fe47121-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-r7zl5\" (UID: \"2b49d636-77b9-487e-aa76-05a41fe47121\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-r7zl5"
Jul 14 11:46:24.954913 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:24.954602    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b49d636-77b9-487e-aa76-05a41fe47121-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-r7zl5\" (UID: \"2b49d636-77b9-487e-aa76-05a41fe47121\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-r7zl5"
Jul 14 11:46:24.956490 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:24.956468    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b49d636-77b9-487e-aa76-05a41fe47121-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-r7zl5\" (UID: \"2b49d636-77b9-487e-aa76-05a41fe47121\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-r7zl5"
Jul 14 11:46:24.956610 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:24.956537    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b49d636-77b9-487e-aa76-05a41fe47121-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-r7zl5\" (UID: \"2b49d636-77b9-487e-aa76-05a41fe47121\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-r7zl5"
Jul 14 11:46:24.964916 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:24.964886    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b49d636-77b9-487e-aa76-05a41fe47121-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-r7zl5\" (UID: \"2b49d636-77b9-487e-aa76-05a41fe47121\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-r7zl5"
Jul 14 11:46:24.965024 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:24.965007    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmjbl\" (UniqueName: \"kubernetes.io/projected/2b49d636-77b9-487e-aa76-05a41fe47121-kube-api-access-mmjbl\") pod \"maas-default-gateway-openshift-default-56654dcf7-r7zl5\" (UID: \"2b49d636-77b9-487e-aa76-05a41fe47121\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-r7zl5"
Jul 14 11:46:25.126052 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:25.125960    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-r7zl5"
Jul 14 11:46:25.302099 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:25.302073    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-r7zl5"]
Jul 14 11:46:25.304244 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:46:25.304214    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b49d636_77b9_487e_aa76_05a41fe47121.slice/crio-a23dfec3e0f45e068a29637e21e9f4620f3284b33972d1971c6f3b37309ccf22 WatchSource:0}: Error finding container a23dfec3e0f45e068a29637e21e9f4620f3284b33972d1971c6f3b37309ccf22: Status 404 returned error can't find the container with id a23dfec3e0f45e068a29637e21e9f4620f3284b33972d1971c6f3b37309ccf22
Jul 14 11:46:25.353284 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:25.353246    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-r7zl5" event={"ID":"2b49d636-77b9-487e-aa76-05a41fe47121","Type":"ContainerStarted","Data":"a23dfec3e0f45e068a29637e21e9f4620f3284b33972d1971c6f3b37309ccf22"}
Jul 14 11:46:25.571758 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:25.571717    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:46:28.101920 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:28.101878    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 14 11:46:28.102195 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:28.101963    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 14 11:46:28.102195 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:28.101999    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 14 11:46:28.369732 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:28.369645    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-r7zl5" event={"ID":"2b49d636-77b9-487e-aa76-05a41fe47121","Type":"ContainerStarted","Data":"f551d38fc8b3fda583d0a37e3b066574c66b59d15a9618211b19bfce818b0dcf"}
Jul 14 11:46:28.399706 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:28.399656    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-r7zl5" podStartSLOduration=1.60438633 podStartE2EDuration="4.399641088s" totalImagesPullingTime="2.795254758s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 11:46:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 11:46:28.397807101 +0000 UTC m=+539.629616396" watchObservedRunningTime="2026-07-14 11:46:28.399641088 +0000 UTC m=+539.631450432"
Jul 14 11:46:29.126547 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:29.126509    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-r7zl5"
Jul 14 11:46:29.131214 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:29.131189    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-r7zl5"
Jul 14 11:46:29.373921 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:29.373833    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-r7zl5"
Jul 14 11:46:29.374748 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:29.374726    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-r7zl5"
Jul 14 11:46:29.527998 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:29.527946    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:46:29.620128 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:29.620089    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:46:29.743927 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:29.743887    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:46:29.948289 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:29.948253    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 11:46:29.952867 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:29.952843    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"
Jul 14 11:46:29.955549 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:29.955526    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 14 11:46:29.963090 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:29.963066    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 11:46:29.997427 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:29.997348    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 11:46:30.004927 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:30.004896    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/92a411dc-0055-4a4c-8302-5ffc7ceff6a0-config-file\") pod \"limitador-limitador-54d6d988d5-7pqvf\" (UID: \"92a411dc-0055-4a4c-8302-5ffc7ceff6a0\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"
Jul 14 11:46:30.005094 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:30.004943    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwng8\" (UniqueName: \"kubernetes.io/projected/92a411dc-0055-4a4c-8302-5ffc7ceff6a0-kube-api-access-xwng8\") pod \"limitador-limitador-54d6d988d5-7pqvf\" (UID: \"92a411dc-0055-4a4c-8302-5ffc7ceff6a0\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"
Jul 14 11:46:30.106019 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:30.105968    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwng8\" (UniqueName: \"kubernetes.io/projected/92a411dc-0055-4a4c-8302-5ffc7ceff6a0-kube-api-access-xwng8\") pod \"limitador-limitador-54d6d988d5-7pqvf\" (UID: \"92a411dc-0055-4a4c-8302-5ffc7ceff6a0\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"
Jul 14 11:46:30.106204 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:30.106096    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/92a411dc-0055-4a4c-8302-5ffc7ceff6a0-config-file\") pod \"limitador-limitador-54d6d988d5-7pqvf\" (UID: \"92a411dc-0055-4a4c-8302-5ffc7ceff6a0\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"
Jul 14 11:46:30.106780 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:30.106760    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/92a411dc-0055-4a4c-8302-5ffc7ceff6a0-config-file\") pod \"limitador-limitador-54d6d988d5-7pqvf\" (UID: \"92a411dc-0055-4a4c-8302-5ffc7ceff6a0\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"
Jul 14 11:46:30.116644 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:30.116620    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwng8\" (UniqueName: \"kubernetes.io/projected/92a411dc-0055-4a4c-8302-5ffc7ceff6a0-kube-api-access-xwng8\") pod \"limitador-limitador-54d6d988d5-7pqvf\" (UID: \"92a411dc-0055-4a4c-8302-5ffc7ceff6a0\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"
Jul 14 11:46:30.263131 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:30.263038    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"
Jul 14 11:46:30.413269 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:30.413233    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 11:46:30.414547 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:30.414519    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:46:30.417680 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:46:30.417642    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92a411dc_0055_4a4c_8302_5ffc7ceff6a0.slice/crio-d3a9d51294be572f44dfdec9ea493255b81b5543689d2b64bce1dd7b4025f768 WatchSource:0}: Error finding container d3a9d51294be572f44dfdec9ea493255b81b5543689d2b64bce1dd7b4025f768: Status 404 returned error can't find the container with id d3a9d51294be572f44dfdec9ea493255b81b5543689d2b64bce1dd7b4025f768
Jul 14 11:46:30.514145 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:30.514064    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:46:30.673098 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:30.673066    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-h9v2t"]
Jul 14 11:46:30.677490 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:30.677468    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-h9v2t"
Jul 14 11:46:30.680418 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:30.680396    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-rnk6k\""
Jul 14 11:46:30.687404 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:30.687381    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-h9v2t"]
Jul 14 11:46:30.811796 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:30.811706    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hrc5\" (UniqueName: \"kubernetes.io/projected/69a96690-f3d5-4705-89ef-9276645d38ab-kube-api-access-2hrc5\") pod \"authorino-75d556bd77-h9v2t\" (UID: \"69a96690-f3d5-4705-89ef-9276645d38ab\") " pod="kuadrant-system/authorino-75d556bd77-h9v2t"
Jul 14 11:46:30.912766 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:30.912724    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hrc5\" (UniqueName: \"kubernetes.io/projected/69a96690-f3d5-4705-89ef-9276645d38ab-kube-api-access-2hrc5\") pod \"authorino-75d556bd77-h9v2t\" (UID: \"69a96690-f3d5-4705-89ef-9276645d38ab\") " pod="kuadrant-system/authorino-75d556bd77-h9v2t"
Jul 14 11:46:30.927605 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:30.925866    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hrc5\" (UniqueName: \"kubernetes.io/projected/69a96690-f3d5-4705-89ef-9276645d38ab-kube-api-access-2hrc5\") pod \"authorino-75d556bd77-h9v2t\" (UID: \"69a96690-f3d5-4705-89ef-9276645d38ab\") " pod="kuadrant-system/authorino-75d556bd77-h9v2t"
Jul 14 11:46:30.987393 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:30.987345    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-h9v2t"
Jul 14 11:46:31.184102 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:31.183996    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-h9v2t"]
Jul 14 11:46:31.187980 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:46:31.187946    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod69a96690_f3d5_4705_89ef_9276645d38ab.slice/crio-23e63ad20017092e4003f40d35a8729f0f094b1fb322d5af587dd3a2e01b2fac WatchSource:0}: Error finding container 23e63ad20017092e4003f40d35a8729f0f094b1fb322d5af587dd3a2e01b2fac: Status 404 returned error can't find the container with id 23e63ad20017092e4003f40d35a8729f0f094b1fb322d5af587dd3a2e01b2fac
Jul 14 11:46:31.385555 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:31.385459    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-7pqvf" event={"ID":"92a411dc-0055-4a4c-8302-5ffc7ceff6a0","Type":"ContainerStarted","Data":"d3a9d51294be572f44dfdec9ea493255b81b5543689d2b64bce1dd7b4025f768"}
Jul 14 11:46:31.386837 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:31.386803    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-h9v2t" event={"ID":"69a96690-f3d5-4705-89ef-9276645d38ab","Type":"ContainerStarted","Data":"23e63ad20017092e4003f40d35a8729f0f094b1fb322d5af587dd3a2e01b2fac"}
Jul 14 11:46:33.399691 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:33.399603    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-7pqvf" event={"ID":"92a411dc-0055-4a4c-8302-5ffc7ceff6a0","Type":"ContainerStarted","Data":"83c2222174d2dd6b77af3ed0a1db1d81fc197bd22d367b9b2de9ef98d44e7de2"}
Jul 14 11:46:33.400143 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:33.399714    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"
Jul 14 11:46:33.422509 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:33.422451    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-7pqvf" podStartSLOduration=1.795304374 podStartE2EDuration="4.422433437s" totalImagesPullingTime="2.627129063s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 11:46:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 11:46:33.420857337 +0000 UTC m=+544.652666658" watchObservedRunningTime="2026-07-14 11:46:33.422433437 +0000 UTC m=+544.654242733"
Jul 14 11:46:36.415167 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:36.415132    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-h9v2t" event={"ID":"69a96690-f3d5-4705-89ef-9276645d38ab","Type":"ContainerStarted","Data":"494e6795645d6deab6bc91e990a97bff977db30e3e71cd4d1de66194e2d54ff0"}
Jul 14 11:46:36.433602 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:36.433529    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-h9v2t" podStartSLOduration=2.165624933 podStartE2EDuration="6.433513557s" totalImagesPullingTime="4.267888624s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 11:46:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 11:46:36.433283895 +0000 UTC m=+547.665093191" watchObservedRunningTime="2026-07-14 11:46:36.433513557 +0000 UTC m=+547.665322852"
Jul 14 11:46:44.407130 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:44.407094    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"
Jul 14 11:46:45.581541 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:45.581504    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:46:45.682561 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:45.682526    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:46:46.070600 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:46.070546    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:46:58.215489 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:58.215450    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:46:58.571591 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:46:58.571490    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:47:03.134127 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:47:03.134085    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-h9v2t"]
Jul 14 11:47:03.134683 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:47:03.134300    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-h9v2t" podUID="69a96690-f3d5-4705-89ef-9276645d38ab" containerName="authorino" containerID="cri-o://494e6795645d6deab6bc91e990a97bff977db30e3e71cd4d1de66194e2d54ff0" gracePeriod=30
Jul 14 11:47:03.390296 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:47:03.390230    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-h9v2t"
Jul 14 11:47:03.499678 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:47:03.499640    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2hrc5\" (UniqueName: \"kubernetes.io/projected/69a96690-f3d5-4705-89ef-9276645d38ab-kube-api-access-2hrc5\") pod \"69a96690-f3d5-4705-89ef-9276645d38ab\" (UID: \"69a96690-f3d5-4705-89ef-9276645d38ab\") "
Jul 14 11:47:03.501741 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:47:03.501702    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69a96690-f3d5-4705-89ef-9276645d38ab-kube-api-access-2hrc5" (OuterVolumeSpecName: "kube-api-access-2hrc5") pod "69a96690-f3d5-4705-89ef-9276645d38ab" (UID: "69a96690-f3d5-4705-89ef-9276645d38ab"). InnerVolumeSpecName "kube-api-access-2hrc5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 11:47:03.542949 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:47:03.542910    2551 generic.go:356] "Generic (PLEG): container finished" podID="69a96690-f3d5-4705-89ef-9276645d38ab" containerID="494e6795645d6deab6bc91e990a97bff977db30e3e71cd4d1de66194e2d54ff0" exitCode=0
Jul 14 11:47:03.543106 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:47:03.542950    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-h9v2t"
Jul 14 11:47:03.543106 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:47:03.542958    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-h9v2t" event={"ID":"69a96690-f3d5-4705-89ef-9276645d38ab","Type":"ContainerDied","Data":"494e6795645d6deab6bc91e990a97bff977db30e3e71cd4d1de66194e2d54ff0"}
Jul 14 11:47:03.543106 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:47:03.542991    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-h9v2t" event={"ID":"69a96690-f3d5-4705-89ef-9276645d38ab","Type":"ContainerDied","Data":"23e63ad20017092e4003f40d35a8729f0f094b1fb322d5af587dd3a2e01b2fac"}
Jul 14 11:47:03.543106 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:47:03.543007    2551 scope.go:117] "RemoveContainer" containerID="494e6795645d6deab6bc91e990a97bff977db30e3e71cd4d1de66194e2d54ff0"
Jul 14 11:47:03.551826 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:47:03.551807    2551 scope.go:117] "RemoveContainer" containerID="494e6795645d6deab6bc91e990a97bff977db30e3e71cd4d1de66194e2d54ff0"
Jul 14 11:47:03.552124 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:47:03.552101    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"494e6795645d6deab6bc91e990a97bff977db30e3e71cd4d1de66194e2d54ff0\": container with ID starting with 494e6795645d6deab6bc91e990a97bff977db30e3e71cd4d1de66194e2d54ff0 not found: ID does not exist" containerID="494e6795645d6deab6bc91e990a97bff977db30e3e71cd4d1de66194e2d54ff0"
Jul 14 11:47:03.552197 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:47:03.552135    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"494e6795645d6deab6bc91e990a97bff977db30e3e71cd4d1de66194e2d54ff0"} err="failed to get container status \"494e6795645d6deab6bc91e990a97bff977db30e3e71cd4d1de66194e2d54ff0\": rpc error: code = NotFound desc = could not find container \"494e6795645d6deab6bc91e990a97bff977db30e3e71cd4d1de66194e2d54ff0\": container with ID starting with 494e6795645d6deab6bc91e990a97bff977db30e3e71cd4d1de66194e2d54ff0 not found: ID does not exist"
Jul 14 11:47:03.565616 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:47:03.565564    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-h9v2t"]
Jul 14 11:47:03.569985 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:47:03.569961    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-h9v2t"]
Jul 14 11:47:03.600481 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:47:03.600443    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hrc5\" (UniqueName: \"kubernetes.io/projected/69a96690-f3d5-4705-89ef-9276645d38ab-kube-api-access-2hrc5\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 11:47:05.191117 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:47:05.191083    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69a96690-f3d5-4705-89ef-9276645d38ab" path="/var/lib/kubelet/pods/69a96690-f3d5-4705-89ef-9276645d38ab/volumes"
Jul 14 11:47:07.984371 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:47:07.984334    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:47:08.094448 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:47:08.094412    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:47:08.181204 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:47:08.181166    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:47:11.424108 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:47:11.424071    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:47:11.523611 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:47:11.523546    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:47:12.462567 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:47:12.462524    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:47:12.564607 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:47:12.564546    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:47:15.449532 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:47:15.449494    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:47:29.081998 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:47:29.081972    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cp57b_62c98a11-ece4-4497-99e7-0840a7626a56/ovn-acl-logging/0.log"
Jul 14 11:47:29.083774 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:47:29.083756    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cp57b_62c98a11-ece4-4497-99e7-0840a7626a56/ovn-acl-logging/0.log"
Jul 14 11:47:32.664087 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:47:32.664051    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:47:39.889601 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:47:39.889541    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:47:41.455786 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:47:41.455754    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:47:41.547905 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:47:41.547872    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:47:42.071463 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:47:42.071424    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:47:42.312304 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:47:42.312272    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:47:42.686325 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:47:42.686292    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:47:42.782321 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:47:42.782285    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:47:42.903261 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:47:42.903229    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 11:47:50.769885 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:47:50.769849    2551 generic.go:356] "Generic (PLEG): container finished" podID="2b49d636-77b9-487e-aa76-05a41fe47121" containerID="f551d38fc8b3fda583d0a37e3b066574c66b59d15a9618211b19bfce818b0dcf" exitCode=0
Jul 14 11:47:50.770399 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:47:50.769923    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-r7zl5" event={"ID":"2b49d636-77b9-487e-aa76-05a41fe47121","Type":"ContainerDied","Data":"f551d38fc8b3fda583d0a37e3b066574c66b59d15a9618211b19bfce818b0dcf"}
Jul 14 11:47:50.770399 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:47:50.770395    2551 scope.go:117] "RemoveContainer" containerID="f551d38fc8b3fda583d0a37e3b066574c66b59d15a9618211b19bfce818b0dcf"
Jul 14 11:47:50.771390 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:47:50.771360    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 14 11:47:50.771482 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:47:50.771448    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 14 11:47:50.771541 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:47:50.771495    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 14 11:47:51.776246 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:47:51.776213    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-r7zl5" event={"ID":"2b49d636-77b9-487e-aa76-05a41fe47121","Type":"ContainerStarted","Data":"420af86684e20e4e6a5723142516468749e73d31f06caeef0ec9eb110a467a49"}
Jul 14 11:47:52.126722 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:47:52.126631    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-r7zl5"
Jul 14 11:47:52.126923 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:47:52.126896    2551 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-r7zl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body=
Jul 14 11:47:52.126993 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:47:52.126972    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-r7zl5" podUID="2b49d636-77b9-487e-aa76-05a41fe47121" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused"
Jul 14 11:47:52.149308 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:47:52.149269    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:47:52.247468 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:47:52.247390    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:47:53.127170 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:47:53.127130    2551 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-r7zl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body=
Jul 14 11:47:53.127686 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:47:53.127199    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-r7zl5" podUID="2b49d636-77b9-487e-aa76-05a41fe47121" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused"
Jul 14 11:47:54.126486 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:47:54.126447    2551 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-r7zl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body=
Jul 14 11:47:54.126825 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:47:54.126784    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-r7zl5" podUID="2b49d636-77b9-487e-aa76-05a41fe47121" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused"
Jul 14 11:47:55.126985 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:47:55.126943    2551 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-r7zl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body=
Jul 14 11:47:55.127601 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:47:55.127545    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-r7zl5" podUID="2b49d636-77b9-487e-aa76-05a41fe47121" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused"
Jul 14 11:47:56.130827 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:47:56.130796    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-r7zl5"
Jul 14 11:47:56.131229 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:47:56.131168    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-r7zl5"
Jul 14 11:47:56.131999 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:47:56.131978    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-r7zl5"
Jul 14 11:47:56.775988 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:47:56.775954    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9nnlc"]
Jul 14 11:47:56.776257 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:47:56.776246    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69a96690-f3d5-4705-89ef-9276645d38ab" containerName="authorino"
Jul 14 11:47:56.776301 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:47:56.776259    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="69a96690-f3d5-4705-89ef-9276645d38ab" containerName="authorino"
Jul 14 11:47:56.776339 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:47:56.776326    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="69a96690-f3d5-4705-89ef-9276645d38ab" containerName="authorino"
Jul 14 11:47:56.779335 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:47:56.779312    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9nnlc"
Jul 14 11:47:56.781993 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:47:56.781949    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-8qpv8\""
Jul 14 11:47:56.782118 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:47:56.781954    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 14 11:47:56.782118 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:47:56.782025    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 14 11:47:56.782118 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:47:56.782035    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Jul 14 11:47:56.791854 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:47:56.791825    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9nnlc"]
Jul 14 11:47:56.809587 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:47:56.809563    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:47:56.844925 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:47:56.844903    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74e33bbf-9897-4a16-a7e5-5a2bfcc7518a-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9nnlc\" (UID: \"74e33bbf-9897-4a16-a7e5-5a2bfcc7518a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9nnlc"
Jul 14 11:47:56.845027 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:47:56.844957    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nbck\" (UniqueName: \"kubernetes.io/projected/74e33bbf-9897-4a16-a7e5-5a2bfcc7518a-kube-api-access-6nbck\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9nnlc\" (UID: \"74e33bbf-9897-4a16-a7e5-5a2bfcc7518a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9nnlc"
Jul 14 11:47:56.845080 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:47:56.845049    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74e33bbf-9897-4a16-a7e5-5a2bfcc7518a-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9nnlc\" (UID: \"74e33bbf-9897-4a16-a7e5-5a2bfcc7518a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9nnlc"
Jul 14 11:47:56.845080 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:47:56.845077    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74e33bbf-9897-4a16-a7e5-5a2bfcc7518a-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9nnlc\" (UID: \"74e33bbf-9897-4a16-a7e5-5a2bfcc7518a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9nnlc"
Jul 14 11:47:56.845161 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:47:56.845111    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74e33bbf-9897-4a16-a7e5-5a2bfcc7518a-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9nnlc\" (UID: \"74e33bbf-9897-4a16-a7e5-5a2bfcc7518a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9nnlc"
Jul 14 11:47:56.845161 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:47:56.845127    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74e33bbf-9897-4a16-a7e5-5a2bfcc7518a-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9nnlc\" (UID: \"74e33bbf-9897-4a16-a7e5-5a2bfcc7518a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9nnlc"
Jul 14 11:47:56.945480 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:47:56.945443    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74e33bbf-9897-4a16-a7e5-5a2bfcc7518a-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9nnlc\" (UID: \"74e33bbf-9897-4a16-a7e5-5a2bfcc7518a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9nnlc"
Jul 14 11:47:56.945679 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:47:56.945497    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nbck\" (UniqueName: \"kubernetes.io/projected/74e33bbf-9897-4a16-a7e5-5a2bfcc7518a-kube-api-access-6nbck\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9nnlc\" (UID: \"74e33bbf-9897-4a16-a7e5-5a2bfcc7518a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9nnlc"
Jul 14 11:47:56.945679 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:47:56.945538    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74e33bbf-9897-4a16-a7e5-5a2bfcc7518a-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9nnlc\" (UID: \"74e33bbf-9897-4a16-a7e5-5a2bfcc7518a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9nnlc"
Jul 14 11:47:56.945679 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:47:56.945555    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74e33bbf-9897-4a16-a7e5-5a2bfcc7518a-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9nnlc\" (UID: \"74e33bbf-9897-4a16-a7e5-5a2bfcc7518a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9nnlc"
Jul 14 11:47:56.945679 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:47:56.945668    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74e33bbf-9897-4a16-a7e5-5a2bfcc7518a-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9nnlc\" (UID: \"74e33bbf-9897-4a16-a7e5-5a2bfcc7518a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9nnlc"
Jul 14 11:47:56.945965 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:47:56.945695    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74e33bbf-9897-4a16-a7e5-5a2bfcc7518a-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9nnlc\" (UID: \"74e33bbf-9897-4a16-a7e5-5a2bfcc7518a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9nnlc"
Jul 14 11:47:56.945965 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:47:56.945952    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74e33bbf-9897-4a16-a7e5-5a2bfcc7518a-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9nnlc\" (UID: \"74e33bbf-9897-4a16-a7e5-5a2bfcc7518a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9nnlc"
Jul 14 11:47:56.946050 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:47:56.945983    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74e33bbf-9897-4a16-a7e5-5a2bfcc7518a-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9nnlc\" (UID: \"74e33bbf-9897-4a16-a7e5-5a2bfcc7518a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9nnlc"
Jul 14 11:47:56.946050 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:47:56.945996    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74e33bbf-9897-4a16-a7e5-5a2bfcc7518a-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9nnlc\" (UID: \"74e33bbf-9897-4a16-a7e5-5a2bfcc7518a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9nnlc"
Jul 14 11:47:56.947937 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:47:56.947911    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74e33bbf-9897-4a16-a7e5-5a2bfcc7518a-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9nnlc\" (UID: \"74e33bbf-9897-4a16-a7e5-5a2bfcc7518a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9nnlc"
Jul 14 11:47:56.948202 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:47:56.948176    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74e33bbf-9897-4a16-a7e5-5a2bfcc7518a-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9nnlc\" (UID: \"74e33bbf-9897-4a16-a7e5-5a2bfcc7518a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9nnlc"
Jul 14 11:47:56.954355 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:47:56.954325    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nbck\" (UniqueName: \"kubernetes.io/projected/74e33bbf-9897-4a16-a7e5-5a2bfcc7518a-kube-api-access-6nbck\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9nnlc\" (UID: \"74e33bbf-9897-4a16-a7e5-5a2bfcc7518a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9nnlc"
Jul 14 11:47:57.090153 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:47:57.090064    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9nnlc"
Jul 14 11:47:57.224857 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:47:57.224823    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9nnlc"]
Jul 14 11:47:57.228267 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:47:57.228240    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74e33bbf_9897_4a16_a7e5_5a2bfcc7518a.slice/crio-9d41af6c8020aa26744dd8e380409b67fc3de347b56640d1d001979de7407916 WatchSource:0}: Error finding container 9d41af6c8020aa26744dd8e380409b67fc3de347b56640d1d001979de7407916: Status 404 returned error can't find the container with id 9d41af6c8020aa26744dd8e380409b67fc3de347b56640d1d001979de7407916
Jul 14 11:47:57.229896 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:47:57.229875    2551 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 14 11:47:57.377187 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:47:57.377106    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 11:47:57.805873 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:47:57.805836    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9nnlc" event={"ID":"74e33bbf-9897-4a16-a7e5-5a2bfcc7518a","Type":"ContainerStarted","Data":"9d41af6c8020aa26744dd8e380409b67fc3de347b56640d1d001979de7407916"}
Jul 14 11:47:58.812107 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:47:58.812069    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9nnlc" event={"ID":"74e33bbf-9897-4a16-a7e5-5a2bfcc7518a","Type":"ContainerStarted","Data":"06c6d973a79059558c75f3df7fc2da1cc1fd8156bf97b0faf3b4f16d052b5805"}
Jul 14 11:47:58.832783 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:47:58.832701    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9nnlc" podStartSLOduration=1.666225497 podStartE2EDuration="2.83268276s" totalImagesPullingTime="1.166457263s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 11:47:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 11:47:58.831803238 +0000 UTC m=+630.063612548" watchObservedRunningTime="2026-07-14 11:47:58.83268276 +0000 UTC m=+630.064492057"
Jul 14 11:48:06.712537 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:48:06.712504    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:48:07.091239 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:48:07.091153    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9nnlc"
Jul 14 11:48:07.091239 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:48:07.091224    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9nnlc"
Jul 14 11:48:07.103416 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:48:07.103384    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9nnlc"
Jul 14 11:48:07.479390 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:48:07.479352    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 11:48:07.862147 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:48:07.862074    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9nnlc"
Jul 14 11:48:18.898626 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:48:18.898593    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-tz94n"]
Jul 14 11:48:18.902054 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:48:18.902039    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tz94n"
Jul 14 11:48:18.904965 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:48:18.904944    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Jul 14 11:48:18.911764 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:48:18.911731    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-tz94n"]
Jul 14 11:48:18.928096 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:48:18.928056    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:48:19.014850 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:48:19.014824    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d190ee5-ef81-4dad-9661-80b09580d03d-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tz94n\" (UID: \"3d190ee5-ef81-4dad-9661-80b09580d03d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tz94n"
Jul 14 11:48:19.014994 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:48:19.014855    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d190ee5-ef81-4dad-9661-80b09580d03d-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tz94n\" (UID: \"3d190ee5-ef81-4dad-9661-80b09580d03d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tz94n"
Jul 14 11:48:19.014994 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:48:19.014881    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77n2k\" (UniqueName: \"kubernetes.io/projected/3d190ee5-ef81-4dad-9661-80b09580d03d-kube-api-access-77n2k\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tz94n\" (UID: \"3d190ee5-ef81-4dad-9661-80b09580d03d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tz94n"
Jul 14 11:48:19.014994 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:48:19.014906    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d190ee5-ef81-4dad-9661-80b09580d03d-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tz94n\" (UID: \"3d190ee5-ef81-4dad-9661-80b09580d03d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tz94n"
Jul 14 11:48:19.014994 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:48:19.014938    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d190ee5-ef81-4dad-9661-80b09580d03d-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tz94n\" (UID: \"3d190ee5-ef81-4dad-9661-80b09580d03d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tz94n"
Jul 14 11:48:19.014994 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:48:19.014952    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d190ee5-ef81-4dad-9661-80b09580d03d-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tz94n\" (UID: \"3d190ee5-ef81-4dad-9661-80b09580d03d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tz94n"
Jul 14 11:48:19.116206 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:48:19.116167    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d190ee5-ef81-4dad-9661-80b09580d03d-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tz94n\" (UID: \"3d190ee5-ef81-4dad-9661-80b09580d03d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tz94n"
Jul 14 11:48:19.116206 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:48:19.116207    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d190ee5-ef81-4dad-9661-80b09580d03d-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tz94n\" (UID: \"3d190ee5-ef81-4dad-9661-80b09580d03d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tz94n"
Jul 14 11:48:19.116424 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:48:19.116231    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77n2k\" (UniqueName: \"kubernetes.io/projected/3d190ee5-ef81-4dad-9661-80b09580d03d-kube-api-access-77n2k\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tz94n\" (UID: \"3d190ee5-ef81-4dad-9661-80b09580d03d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tz94n"
Jul 14 11:48:19.116424 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:48:19.116251    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d190ee5-ef81-4dad-9661-80b09580d03d-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tz94n\" (UID: \"3d190ee5-ef81-4dad-9661-80b09580d03d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tz94n"
Jul 14 11:48:19.116424 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:48:19.116284    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d190ee5-ef81-4dad-9661-80b09580d03d-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tz94n\" (UID: \"3d190ee5-ef81-4dad-9661-80b09580d03d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tz94n"
Jul 14 11:48:19.116424 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:48:19.116321    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d190ee5-ef81-4dad-9661-80b09580d03d-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tz94n\" (UID: \"3d190ee5-ef81-4dad-9661-80b09580d03d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tz94n"
Jul 14 11:48:19.116700 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:48:19.116679    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d190ee5-ef81-4dad-9661-80b09580d03d-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tz94n\" (UID: \"3d190ee5-ef81-4dad-9661-80b09580d03d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tz94n"
Jul 14 11:48:19.116768 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:48:19.116747    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d190ee5-ef81-4dad-9661-80b09580d03d-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tz94n\" (UID: \"3d190ee5-ef81-4dad-9661-80b09580d03d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tz94n"
Jul 14 11:48:19.116821 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:48:19.116775    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d190ee5-ef81-4dad-9661-80b09580d03d-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tz94n\" (UID: \"3d190ee5-ef81-4dad-9661-80b09580d03d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tz94n"
Jul 14 11:48:19.118593 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:48:19.118564    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d190ee5-ef81-4dad-9661-80b09580d03d-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tz94n\" (UID: \"3d190ee5-ef81-4dad-9661-80b09580d03d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tz94n"
Jul 14 11:48:19.118790 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:48:19.118773    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d190ee5-ef81-4dad-9661-80b09580d03d-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tz94n\" (UID: \"3d190ee5-ef81-4dad-9661-80b09580d03d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tz94n"
Jul 14 11:48:19.125954 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:48:19.125865    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-77n2k\" (UniqueName: \"kubernetes.io/projected/3d190ee5-ef81-4dad-9661-80b09580d03d-kube-api-access-77n2k\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tz94n\" (UID: \"3d190ee5-ef81-4dad-9661-80b09580d03d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tz94n"
Jul 14 11:48:19.212535 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:48:19.212506    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tz94n"
Jul 14 11:48:19.355187 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:48:19.355154    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-tz94n"]
Jul 14 11:48:19.358126 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:48:19.358094    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d190ee5_ef81_4dad_9661_80b09580d03d.slice/crio-6de9572ef1a4e18454667fe9f9165dc042cbb108bea90084f6e97521d75d56a9 WatchSource:0}: Error finding container 6de9572ef1a4e18454667fe9f9165dc042cbb108bea90084f6e97521d75d56a9: Status 404 returned error can't find the container with id 6de9572ef1a4e18454667fe9f9165dc042cbb108bea90084f6e97521d75d56a9
Jul 14 11:48:19.905489 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:48:19.905401    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tz94n" event={"ID":"3d190ee5-ef81-4dad-9661-80b09580d03d","Type":"ContainerStarted","Data":"58a9cb8b31af811d563e7c323eef25139243f942b9ba31d68c719e9da1e0f467"}
Jul 14 11:48:19.905489 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:48:19.905457    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tz94n" event={"ID":"3d190ee5-ef81-4dad-9661-80b09580d03d","Type":"ContainerStarted","Data":"6de9572ef1a4e18454667fe9f9165dc042cbb108bea90084f6e97521d75d56a9"}
Jul 14 11:48:19.926758 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:48:19.926697    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tz94n" podStartSLOduration=1.7414828519999999 podStartE2EDuration="1.926678301s" totalImagesPullingTime="185.195449ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 11:48:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 11:48:19.924340623 +0000 UTC m=+651.156149948" watchObservedRunningTime="2026-07-14 11:48:19.926678301 +0000 UTC m=+651.158487598"
Jul 14 11:48:20.092073 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:48:20.092037    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 11:48:21.586664 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:48:21.586630    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:48:26.682710 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:48:26.682675    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 11:48:29.213076 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:48:29.213043    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tz94n"
Jul 14 11:48:29.213495 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:48:29.213112    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tz94n"
Jul 14 11:48:29.225696 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:48:29.225673    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tz94n"
Jul 14 11:48:29.968506 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:48:29.968475    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tz94n"
Jul 14 11:48:43.708358 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:48:43.708322    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:48:43.794016 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:48:43.793976    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:48:43.941634 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:48:43.941571    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:48:45.782367 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:48:45.782322    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 11:48:54.426833 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:48:54.426792    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:48:55.746606 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:48:55.746554    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:48:57.782835 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:48:57.782791    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 11:49:00.849775 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:49:00.849738    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:49:48.935906 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:49:48.935803    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:49:49.026323 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:49:49.026287    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:49:50.187371 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:49:50.187338    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:49:50.320384 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:49:50.320349    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:50:20.617884 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:50:20.617848    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 11:50:30.218568 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:50:30.218530    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 11:50:41.695048 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:50:41.695015    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 11:50:49.599447 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:50:49.599411    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 11:51:00.910619 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:51:00.910566    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 11:51:02.567907 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:51:02.567865    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:51:03.951854 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:51:03.951813    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:51:27.387651 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:51:27.387612    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 11:51:39.900946 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:51:39.900903    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 11:51:44.775693 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:51:44.775645    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:51:44.956939 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:51:44.956903    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:51:45.074018 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:51:45.073919    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:51:52.306292 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:51:52.306256    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:51:52.444964 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:51:52.444932    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:51:52.549826 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:51:52.549793    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:51:52.703105 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:51:52.703058    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:51:59.817063 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:51:59.817036    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:51:59.971808 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:51:59.971773    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:52:17.195502 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:52:17.195451    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 11:52:27.391122 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:52:27.391040    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 11:52:29.108119 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:52:29.108094    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cp57b_62c98a11-ece4-4497-99e7-0840a7626a56/ovn-acl-logging/0.log"
Jul 14 11:52:29.108562 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:52:29.108443    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cp57b_62c98a11-ece4-4497-99e7-0840a7626a56/ovn-acl-logging/0.log"
Jul 14 11:52:36.290451 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:52:36.290419    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 11:52:43.500593 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:52:43.500553    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 11:52:54.626231 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:52:54.626196    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 11:53:05.599524 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:53:05.599487    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 11:53:16.734248 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:53:16.734206    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 11:53:26.507750 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:53:26.507712    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 11:53:52.424021 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:53:52.423926    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 11:54:03.690380 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:54:03.690344    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 11:54:17.666220 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:54:17.666185    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:54:26.488878 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:54:26.488840    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 11:54:37.219331 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:54:37.219300    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 11:54:52.282653 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:54:52.282617    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 11:55:03.583551 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:55:03.583512    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 11:55:12.196730 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:55:12.196645    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:55:13.192340 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:55:13.192311    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 11:55:22.487917 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:55:22.487884    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 11:55:31.891192 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:55:31.891157    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 11:55:58.968992 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:55:58.968955    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-j7b55"]
Jul 14 11:55:58.969534 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:55:58.969155    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-j7b55" podUID="4c16658c-2c58-4e12-9bca-cbb43c9e970e" containerName="manager" containerID="cri-o://a8eb4a46c9046a40183b328e4451b682d22c48dd5e4666628d77a97e20b0eedf" gracePeriod=10
Jul 14 11:56:01.418056 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:56:01.418031    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-j7b55"
Jul 14 11:56:01.544976 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:56:01.544894    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4c16658c-2c58-4e12-9bca-cbb43c9e970e-extensions-socket-volume\") pod \"4c16658c-2c58-4e12-9bca-cbb43c9e970e\" (UID: \"4c16658c-2c58-4e12-9bca-cbb43c9e970e\") "
Jul 14 11:56:01.544976 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:56:01.544944    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v5qmh\" (UniqueName: \"kubernetes.io/projected/4c16658c-2c58-4e12-9bca-cbb43c9e970e-kube-api-access-v5qmh\") pod \"4c16658c-2c58-4e12-9bca-cbb43c9e970e\" (UID: \"4c16658c-2c58-4e12-9bca-cbb43c9e970e\") "
Jul 14 11:56:01.545363 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:56:01.545337    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c16658c-2c58-4e12-9bca-cbb43c9e970e-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "4c16658c-2c58-4e12-9bca-cbb43c9e970e" (UID: "4c16658c-2c58-4e12-9bca-cbb43c9e970e"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 11:56:01.546995 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:56:01.546975    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c16658c-2c58-4e12-9bca-cbb43c9e970e-kube-api-access-v5qmh" (OuterVolumeSpecName: "kube-api-access-v5qmh") pod "4c16658c-2c58-4e12-9bca-cbb43c9e970e" (UID: "4c16658c-2c58-4e12-9bca-cbb43c9e970e"). InnerVolumeSpecName "kube-api-access-v5qmh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 11:56:01.646346 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:56:01.646300    2551 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4c16658c-2c58-4e12-9bca-cbb43c9e970e-extensions-socket-volume\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 11:56:01.646346 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:56:01.646335    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5qmh\" (UniqueName: \"kubernetes.io/projected/4c16658c-2c58-4e12-9bca-cbb43c9e970e-kube-api-access-v5qmh\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 11:56:01.991747 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:56:01.991714    2551 generic.go:356] "Generic (PLEG): container finished" podID="4c16658c-2c58-4e12-9bca-cbb43c9e970e" containerID="a8eb4a46c9046a40183b328e4451b682d22c48dd5e4666628d77a97e20b0eedf" exitCode=0
Jul 14 11:56:01.991747 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:56:01.991749    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-j7b55"
Jul 14 11:56:01.991952 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:56:01.991784    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-j7b55" event={"ID":"4c16658c-2c58-4e12-9bca-cbb43c9e970e","Type":"ContainerDied","Data":"a8eb4a46c9046a40183b328e4451b682d22c48dd5e4666628d77a97e20b0eedf"}
Jul 14 11:56:01.991952 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:56:01.991832    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-j7b55" event={"ID":"4c16658c-2c58-4e12-9bca-cbb43c9e970e","Type":"ContainerDied","Data":"b0640ed931658218695b2c5f65aeffeb230c3566a9478df3a52b1c2acbb285ca"}
Jul 14 11:56:01.991952 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:56:01.991853    2551 scope.go:117] "RemoveContainer" containerID="a8eb4a46c9046a40183b328e4451b682d22c48dd5e4666628d77a97e20b0eedf"
Jul 14 11:56:02.001330 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:56:02.001311    2551 scope.go:117] "RemoveContainer" containerID="a8eb4a46c9046a40183b328e4451b682d22c48dd5e4666628d77a97e20b0eedf"
Jul 14 11:56:02.001563 ip-10-0-136-173 kubenswrapper[2551]: E0714 11:56:02.001543    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8eb4a46c9046a40183b328e4451b682d22c48dd5e4666628d77a97e20b0eedf\": container with ID starting with a8eb4a46c9046a40183b328e4451b682d22c48dd5e4666628d77a97e20b0eedf not found: ID does not exist" containerID="a8eb4a46c9046a40183b328e4451b682d22c48dd5e4666628d77a97e20b0eedf"
Jul 14 11:56:02.001648 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:56:02.001569    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8eb4a46c9046a40183b328e4451b682d22c48dd5e4666628d77a97e20b0eedf"} err="failed to get container status \"a8eb4a46c9046a40183b328e4451b682d22c48dd5e4666628d77a97e20b0eedf\": rpc error: code = NotFound desc = could not find container \"a8eb4a46c9046a40183b328e4451b682d22c48dd5e4666628d77a97e20b0eedf\": container with ID starting with a8eb4a46c9046a40183b328e4451b682d22c48dd5e4666628d77a97e20b0eedf not found: ID does not exist"
Jul 14 11:56:02.018806 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:56:02.018781    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-j7b55"]
Jul 14 11:56:02.024833 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:56:02.024809    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-j7b55"]
Jul 14 11:56:03.190835 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:56:03.190805    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c16658c-2c58-4e12-9bca-cbb43c9e970e" path="/var/lib/kubelet/pods/4c16658c-2c58-4e12-9bca-cbb43c9e970e/volumes"
Jul 14 11:56:16.934950 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:56:16.934916    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:57:00.303118 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:57:00.303079    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fnhg4"]
Jul 14 11:57:00.303619 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:57:00.303419    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c16658c-2c58-4e12-9bca-cbb43c9e970e" containerName="manager"
Jul 14 11:57:00.303619 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:57:00.303430    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c16658c-2c58-4e12-9bca-cbb43c9e970e" containerName="manager"
Jul 14 11:57:00.303619 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:57:00.303496    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c16658c-2c58-4e12-9bca-cbb43c9e970e" containerName="manager"
Jul 14 11:57:00.306327 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:57:00.306311    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fnhg4"
Jul 14 11:57:00.309322 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:57:00.309303    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-lx5tn\""
Jul 14 11:57:00.321389 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:57:00.321367    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fnhg4"]
Jul 14 11:57:00.356669 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:57:00.356634    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drcm5\" (UniqueName: \"kubernetes.io/projected/85a8f743-50a7-4e22-8fd5-585aa4c9d2d9-kube-api-access-drcm5\") pod \"kuadrant-operator-controller-manager-86878c769f-fnhg4\" (UID: \"85a8f743-50a7-4e22-8fd5-585aa4c9d2d9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fnhg4"
Jul 14 11:57:00.356862 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:57:00.356743    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/85a8f743-50a7-4e22-8fd5-585aa4c9d2d9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-fnhg4\" (UID: \"85a8f743-50a7-4e22-8fd5-585aa4c9d2d9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fnhg4"
Jul 14 11:57:00.457798 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:57:00.457761    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/85a8f743-50a7-4e22-8fd5-585aa4c9d2d9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-fnhg4\" (UID: \"85a8f743-50a7-4e22-8fd5-585aa4c9d2d9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fnhg4"
Jul 14 11:57:00.457960 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:57:00.457907    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drcm5\" (UniqueName: \"kubernetes.io/projected/85a8f743-50a7-4e22-8fd5-585aa4c9d2d9-kube-api-access-drcm5\") pod \"kuadrant-operator-controller-manager-86878c769f-fnhg4\" (UID: \"85a8f743-50a7-4e22-8fd5-585aa4c9d2d9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fnhg4"
Jul 14 11:57:00.458180 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:57:00.458159    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/85a8f743-50a7-4e22-8fd5-585aa4c9d2d9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-fnhg4\" (UID: \"85a8f743-50a7-4e22-8fd5-585aa4c9d2d9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fnhg4"
Jul 14 11:57:00.468456 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:57:00.468419    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drcm5\" (UniqueName: \"kubernetes.io/projected/85a8f743-50a7-4e22-8fd5-585aa4c9d2d9-kube-api-access-drcm5\") pod \"kuadrant-operator-controller-manager-86878c769f-fnhg4\" (UID: \"85a8f743-50a7-4e22-8fd5-585aa4c9d2d9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fnhg4"
Jul 14 11:57:00.615867 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:57:00.615778    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fnhg4"
Jul 14 11:57:00.757790 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:57:00.757765    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fnhg4"]
Jul 14 11:57:00.759987 ip-10-0-136-173 kubenswrapper[2551]: W0714 11:57:00.759962    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85a8f743_50a7_4e22_8fd5_585aa4c9d2d9.slice/crio-c4bb0035d118dacc8e761442bc91a295839f4f8e7e9a6638d53fa1f68d21b4cf WatchSource:0}: Error finding container c4bb0035d118dacc8e761442bc91a295839f4f8e7e9a6638d53fa1f68d21b4cf: Status 404 returned error can't find the container with id c4bb0035d118dacc8e761442bc91a295839f4f8e7e9a6638d53fa1f68d21b4cf
Jul 14 11:57:01.265480 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:57:01.265443    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fnhg4" event={"ID":"85a8f743-50a7-4e22-8fd5-585aa4c9d2d9","Type":"ContainerStarted","Data":"98d6042c3813a4c718824a3003aae6abf0a7ea26bf5ff7390676811e2013a9fa"}
Jul 14 11:57:01.265683 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:57:01.265493    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fnhg4" event={"ID":"85a8f743-50a7-4e22-8fd5-585aa4c9d2d9","Type":"ContainerStarted","Data":"c4bb0035d118dacc8e761442bc91a295839f4f8e7e9a6638d53fa1f68d21b4cf"}
Jul 14 11:57:01.265683 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:57:01.265530    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fnhg4"
Jul 14 11:57:01.294147 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:57:01.294096    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fnhg4" podStartSLOduration=1.2940802869999999 podStartE2EDuration="1.294080287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 11:57:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 11:57:01.292666603 +0000 UTC m=+1172.524475902" watchObservedRunningTime="2026-07-14 11:57:01.294080287 +0000 UTC m=+1172.525889560"
Jul 14 11:57:12.272044 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:57:12.272010    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fnhg4"
Jul 14 11:57:20.792347 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:57:20.792312    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 11:57:29.138795 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:57:29.138767    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cp57b_62c98a11-ece4-4497-99e7-0840a7626a56/ovn-acl-logging/0.log"
Jul 14 11:57:29.140500 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:57:29.140478    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cp57b_62c98a11-ece4-4497-99e7-0840a7626a56/ovn-acl-logging/0.log"
Jul 14 11:57:32.090431 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:57:32.090395    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 11:57:40.792995 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:57:40.792956    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 11:57:54.702210 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:57:54.702174    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 11:57:56.579866 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:57:56.579827    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 11:58:04.190155 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:58:04.190118    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 11:58:15.191034 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:58:15.191006    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 11:58:24.309890 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:58:24.309793    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 11:58:34.790207 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:58:34.790174    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 11:58:44.989406 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:58:44.989366    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 11:58:54.904457 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:58:54.904421    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 11:59:05.690308 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:59:05.690271    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 11:59:27.891447 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:59:27.891406    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 11:59:37.800823 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:59:37.800786    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 11:59:47.593511 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:59:47.593479    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 11:59:58.106799 ip-10-0-136-173 kubenswrapper[2551]: I0714 11:59:58.106767    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 12:00:05.221918 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:00:05.221881    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 12:00:15.584526 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:00:15.584487    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 12:00:31.486842 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:00:31.486808    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 12:00:40.302045 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:00:40.302013    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 12:00:47.894385 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:00:47.894351    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 12:00:58.997628 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:00:58.997568    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 12:01:08.496750 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:01:08.496714    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 12:01:18.623737 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:01:18.623700    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:01:20.665983 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:01:20.665901    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:01:21.485066 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:01:21.485030    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 12:01:30.790107 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:01:30.790069    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 12:01:40.295298 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:01:40.295258    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 12:01:51.191098 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:01:51.191066    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 12:02:03.587365 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:02:03.587321    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 12:02:14.089911 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:02:14.089869    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 12:02:25.688150 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:02:25.688112    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 12:02:29.164574 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:02:29.164543    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cp57b_62c98a11-ece4-4497-99e7-0840a7626a56/ovn-acl-logging/0.log"
Jul 14 12:02:29.167418 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:02:29.167396    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cp57b_62c98a11-ece4-4497-99e7-0840a7626a56/ovn-acl-logging/0.log"
Jul 14 12:02:37.995471 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:02:37.995432    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 12:02:50.786895 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:02:50.786807    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 12:02:52.349132 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:02:52.349095    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:02:57.720061 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:02:57.720026    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:03:02.580868 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:03:02.580833    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 12:03:06.748788 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:03:06.748754    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:03:11.175091 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:03:11.175055    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:03:14.336973 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:03:14.336932    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:03:14.437250 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:03:14.437214    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:03:14.821161 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:03:14.821129    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:03:15.115231 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:03:15.115148    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:03:15.309562 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:03:15.309528    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 12:03:21.564225 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:03:21.564190    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:03:21.725081 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:03:21.725040    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:03:21.820189 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:03:21.820110    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:03:22.926491 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:03:22.926450    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:03:23.023415 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:03:23.023387    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:03:23.126275 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:03:23.126240    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ab110be5-84bd7f9468-vbw5g"]
Jul 14 12:03:23.129763 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:03:23.129738    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ab110be5-84bd7f9468-vbw5g"
Jul 14 12:03:23.132596 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:03:23.132547    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 14 12:03:23.133665 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:03:23.133640    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-62k4x\""
Jul 14 12:03:23.133665 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:03:23.133653    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-ab110be5-serving-cert\""
Jul 14 12:03:23.133848 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:03:23.133640    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 14 12:03:23.142174 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:03:23.142147    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ab110be5-84bd7f9468-vbw5g"]
Jul 14 12:03:23.179693 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:03:23.179622    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:03:23.243678 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:03:23.243648    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2dfc09e4-5fe7-4f67-9860-b8ede93c6eaa-maas-api-tls\") pod \"maas-api-e2e-ait-ab110be5-84bd7f9468-vbw5g\" (UID: \"2dfc09e4-5fe7-4f67-9860-b8ede93c6eaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ab110be5-84bd7f9468-vbw5g"
Jul 14 12:03:23.243853 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:03:23.243695    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2dfc09e4-5fe7-4f67-9860-b8ede93c6eaa-openshift-service-ca\") pod \"maas-api-e2e-ait-ab110be5-84bd7f9468-vbw5g\" (UID: \"2dfc09e4-5fe7-4f67-9860-b8ede93c6eaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ab110be5-84bd7f9468-vbw5g"
Jul 14 12:03:23.243853 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:03:23.243750    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcp24\" (UniqueName: \"kubernetes.io/projected/2dfc09e4-5fe7-4f67-9860-b8ede93c6eaa-kube-api-access-wcp24\") pod \"maas-api-e2e-ait-ab110be5-84bd7f9468-vbw5g\" (UID: \"2dfc09e4-5fe7-4f67-9860-b8ede93c6eaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ab110be5-84bd7f9468-vbw5g"
Jul 14 12:03:23.344957 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:03:23.344912    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2dfc09e4-5fe7-4f67-9860-b8ede93c6eaa-maas-api-tls\") pod \"maas-api-e2e-ait-ab110be5-84bd7f9468-vbw5g\" (UID: \"2dfc09e4-5fe7-4f67-9860-b8ede93c6eaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ab110be5-84bd7f9468-vbw5g"
Jul 14 12:03:23.344957 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:03:23.344964    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2dfc09e4-5fe7-4f67-9860-b8ede93c6eaa-openshift-service-ca\") pod \"maas-api-e2e-ait-ab110be5-84bd7f9468-vbw5g\" (UID: \"2dfc09e4-5fe7-4f67-9860-b8ede93c6eaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ab110be5-84bd7f9468-vbw5g"
Jul 14 12:03:23.345216 ip-10-0-136-173 kubenswrapper[2551]: E0714 12:03:23.345061    2551 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-ab110be5-serving-cert: secret "maas-api-e2e-ait-ab110be5-serving-cert" not found
Jul 14 12:03:23.345216 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:03:23.345089    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcp24\" (UniqueName: \"kubernetes.io/projected/2dfc09e4-5fe7-4f67-9860-b8ede93c6eaa-kube-api-access-wcp24\") pod \"maas-api-e2e-ait-ab110be5-84bd7f9468-vbw5g\" (UID: \"2dfc09e4-5fe7-4f67-9860-b8ede93c6eaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ab110be5-84bd7f9468-vbw5g"
Jul 14 12:03:23.345216 ip-10-0-136-173 kubenswrapper[2551]: E0714 12:03:23.345128    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2dfc09e4-5fe7-4f67-9860-b8ede93c6eaa-maas-api-tls podName:2dfc09e4-5fe7-4f67-9860-b8ede93c6eaa nodeName:}" failed. No retries permitted until 2026-07-14 12:03:23.845112662 +0000 UTC m=+1555.076921939 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2dfc09e4-5fe7-4f67-9860-b8ede93c6eaa-maas-api-tls") pod "maas-api-e2e-ait-ab110be5-84bd7f9468-vbw5g" (UID: "2dfc09e4-5fe7-4f67-9860-b8ede93c6eaa") : secret "maas-api-e2e-ait-ab110be5-serving-cert" not found
Jul 14 12:03:23.345724 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:03:23.345696    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2dfc09e4-5fe7-4f67-9860-b8ede93c6eaa-openshift-service-ca\") pod \"maas-api-e2e-ait-ab110be5-84bd7f9468-vbw5g\" (UID: \"2dfc09e4-5fe7-4f67-9860-b8ede93c6eaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ab110be5-84bd7f9468-vbw5g"
Jul 14 12:03:23.356991 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:03:23.356952    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcp24\" (UniqueName: \"kubernetes.io/projected/2dfc09e4-5fe7-4f67-9860-b8ede93c6eaa-kube-api-access-wcp24\") pod \"maas-api-e2e-ait-ab110be5-84bd7f9468-vbw5g\" (UID: \"2dfc09e4-5fe7-4f67-9860-b8ede93c6eaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ab110be5-84bd7f9468-vbw5g"
Jul 14 12:03:23.442999 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:03:23.442968    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:03:23.776307 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:03:23.776223    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:03:23.850179 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:03:23.850140    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2dfc09e4-5fe7-4f67-9860-b8ede93c6eaa-maas-api-tls\") pod \"maas-api-e2e-ait-ab110be5-84bd7f9468-vbw5g\" (UID: \"2dfc09e4-5fe7-4f67-9860-b8ede93c6eaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ab110be5-84bd7f9468-vbw5g"
Jul 14 12:03:23.852547 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:03:23.852518    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2dfc09e4-5fe7-4f67-9860-b8ede93c6eaa-maas-api-tls\") pod \"maas-api-e2e-ait-ab110be5-84bd7f9468-vbw5g\" (UID: \"2dfc09e4-5fe7-4f67-9860-b8ede93c6eaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ab110be5-84bd7f9468-vbw5g"
Jul 14 12:03:23.885708 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:03:23.885677    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:03:24.040082 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:03:24.039978    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ab110be5-84bd7f9468-vbw5g"
Jul 14 12:03:24.180742 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:03:24.180716    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ab110be5-84bd7f9468-vbw5g"]
Jul 14 12:03:24.183022 ip-10-0-136-173 kubenswrapper[2551]: W0714 12:03:24.182995    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dfc09e4_5fe7_4f67_9860_b8ede93c6eaa.slice/crio-e67ad31fada54fb82092de29992b0ea2ffc26877daeb0ee76e99730c57348ac8 WatchSource:0}: Error finding container e67ad31fada54fb82092de29992b0ea2ffc26877daeb0ee76e99730c57348ac8: Status 404 returned error can't find the container with id e67ad31fada54fb82092de29992b0ea2ffc26877daeb0ee76e99730c57348ac8
Jul 14 12:03:24.184309 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:03:24.184291    2551 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 14 12:03:25.024010 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:03:25.023957    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ab110be5-84bd7f9468-vbw5g" event={"ID":"2dfc09e4-5fe7-4f67-9860-b8ede93c6eaa","Type":"ContainerStarted","Data":"e67ad31fada54fb82092de29992b0ea2ffc26877daeb0ee76e99730c57348ac8"}
Jul 14 12:03:25.516879 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:03:25.516790    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:03:26.030289 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:03:26.030254    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ab110be5-84bd7f9468-vbw5g" event={"ID":"2dfc09e4-5fe7-4f67-9860-b8ede93c6eaa","Type":"ContainerStarted","Data":"2de813b4527774b4780a58ce3650e2c0de06b5cc25f02a92541e5c4906f11578"}
Jul 14 12:03:26.030462 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:03:26.030384    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ab110be5-84bd7f9468-vbw5g"
Jul 14 12:03:26.056289 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:03:26.056227    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ab110be5-84bd7f9468-vbw5g" podStartSLOduration=1.374799052 podStartE2EDuration="3.056209106s" totalImagesPullingTime="1.681410054s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 12:03:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 12:03:26.05464866 +0000 UTC m=+1557.286457960" watchObservedRunningTime="2026-07-14 12:03:26.056209106 +0000 UTC m=+1557.288018402"
Jul 14 12:03:31.784992 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:03:31.784946    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 12:03:32.039791 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:03:32.039717    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ab110be5-84bd7f9468-vbw5g"
Jul 14 12:04:28.548431 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:04:28.548347    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ab110be5-84bd7f9468-vbw5g"]
Jul 14 12:04:28.549004 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:04:28.548625    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ab110be5-84bd7f9468-vbw5g" podUID="2dfc09e4-5fe7-4f67-9860-b8ede93c6eaa" containerName="maas-api" containerID="cri-o://2de813b4527774b4780a58ce3650e2c0de06b5cc25f02a92541e5c4906f11578" gracePeriod=30
Jul 14 12:04:28.577831 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:04:28.577801    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:04:28.802917 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:04:28.802848    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ab110be5-84bd7f9468-vbw5g"
Jul 14 12:04:28.941349 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:04:28.941312    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wcp24\" (UniqueName: \"kubernetes.io/projected/2dfc09e4-5fe7-4f67-9860-b8ede93c6eaa-kube-api-access-wcp24\") pod \"2dfc09e4-5fe7-4f67-9860-b8ede93c6eaa\" (UID: \"2dfc09e4-5fe7-4f67-9860-b8ede93c6eaa\") "
Jul 14 12:04:28.941525 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:04:28.941386    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2dfc09e4-5fe7-4f67-9860-b8ede93c6eaa-maas-api-tls\") pod \"2dfc09e4-5fe7-4f67-9860-b8ede93c6eaa\" (UID: \"2dfc09e4-5fe7-4f67-9860-b8ede93c6eaa\") "
Jul 14 12:04:28.941525 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:04:28.941432    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2dfc09e4-5fe7-4f67-9860-b8ede93c6eaa-openshift-service-ca\") pod \"2dfc09e4-5fe7-4f67-9860-b8ede93c6eaa\" (UID: \"2dfc09e4-5fe7-4f67-9860-b8ede93c6eaa\") "
Jul 14 12:04:28.941871 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:04:28.941828    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dfc09e4-5fe7-4f67-9860-b8ede93c6eaa-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2dfc09e4-5fe7-4f67-9860-b8ede93c6eaa" (UID: "2dfc09e4-5fe7-4f67-9860-b8ede93c6eaa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 12:04:28.943606 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:04:28.943556    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2dfc09e4-5fe7-4f67-9860-b8ede93c6eaa-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2dfc09e4-5fe7-4f67-9860-b8ede93c6eaa" (UID: "2dfc09e4-5fe7-4f67-9860-b8ede93c6eaa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 12:04:28.943722 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:04:28.943607    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dfc09e4-5fe7-4f67-9860-b8ede93c6eaa-kube-api-access-wcp24" (OuterVolumeSpecName: "kube-api-access-wcp24") pod "2dfc09e4-5fe7-4f67-9860-b8ede93c6eaa" (UID: "2dfc09e4-5fe7-4f67-9860-b8ede93c6eaa"). InnerVolumeSpecName "kube-api-access-wcp24". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 12:04:29.042219 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:04:29.042178    2551 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2dfc09e4-5fe7-4f67-9860-b8ede93c6eaa-openshift-service-ca\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 12:04:29.042219 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:04:29.042210    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcp24\" (UniqueName: \"kubernetes.io/projected/2dfc09e4-5fe7-4f67-9860-b8ede93c6eaa-kube-api-access-wcp24\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 12:04:29.042219 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:04:29.042221    2551 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2dfc09e4-5fe7-4f67-9860-b8ede93c6eaa-maas-api-tls\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 12:04:29.326957 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:04:29.326919    2551 generic.go:356] "Generic (PLEG): container finished" podID="2dfc09e4-5fe7-4f67-9860-b8ede93c6eaa" containerID="2de813b4527774b4780a58ce3650e2c0de06b5cc25f02a92541e5c4906f11578" exitCode=0
Jul 14 12:04:29.327138 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:04:29.326975    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ab110be5-84bd7f9468-vbw5g"
Jul 14 12:04:29.327138 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:04:29.327002    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ab110be5-84bd7f9468-vbw5g" event={"ID":"2dfc09e4-5fe7-4f67-9860-b8ede93c6eaa","Type":"ContainerDied","Data":"2de813b4527774b4780a58ce3650e2c0de06b5cc25f02a92541e5c4906f11578"}
Jul 14 12:04:29.327138 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:04:29.327067    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ab110be5-84bd7f9468-vbw5g" event={"ID":"2dfc09e4-5fe7-4f67-9860-b8ede93c6eaa","Type":"ContainerDied","Data":"e67ad31fada54fb82092de29992b0ea2ffc26877daeb0ee76e99730c57348ac8"}
Jul 14 12:04:29.327138 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:04:29.327097    2551 scope.go:117] "RemoveContainer" containerID="2de813b4527774b4780a58ce3650e2c0de06b5cc25f02a92541e5c4906f11578"
Jul 14 12:04:29.336111 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:04:29.336091    2551 scope.go:117] "RemoveContainer" containerID="2de813b4527774b4780a58ce3650e2c0de06b5cc25f02a92541e5c4906f11578"
Jul 14 12:04:29.336378 ip-10-0-136-173 kubenswrapper[2551]: E0714 12:04:29.336356    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2de813b4527774b4780a58ce3650e2c0de06b5cc25f02a92541e5c4906f11578\": container with ID starting with 2de813b4527774b4780a58ce3650e2c0de06b5cc25f02a92541e5c4906f11578 not found: ID does not exist" containerID="2de813b4527774b4780a58ce3650e2c0de06b5cc25f02a92541e5c4906f11578"
Jul 14 12:04:29.336454 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:04:29.336391    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2de813b4527774b4780a58ce3650e2c0de06b5cc25f02a92541e5c4906f11578"} err="failed to get container status \"2de813b4527774b4780a58ce3650e2c0de06b5cc25f02a92541e5c4906f11578\": rpc error: code = NotFound desc = could not find container \"2de813b4527774b4780a58ce3650e2c0de06b5cc25f02a92541e5c4906f11578\": container with ID starting with 2de813b4527774b4780a58ce3650e2c0de06b5cc25f02a92541e5c4906f11578 not found: ID does not exist"
Jul 14 12:04:29.351118 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:04:29.348833    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ab110be5-84bd7f9468-vbw5g"]
Jul 14 12:04:29.354216 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:04:29.354190    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ab110be5-84bd7f9468-vbw5g"]
Jul 14 12:04:31.190722 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:04:31.190688    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2dfc09e4-5fe7-4f67-9860-b8ede93c6eaa" path="/var/lib/kubelet/pods/2dfc09e4-5fe7-4f67-9860-b8ede93c6eaa/volumes"
Jul 14 12:04:31.479026 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:04:31.478935    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 12:04:33.133655 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:04:33.133628    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:04:39.833176 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:04:39.833135    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:04:39.911638 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:04:39.911603    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:04:43.406474 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:04:43.406440    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:04:43.786056 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:04:43.786007    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:04:43.886493 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:04:43.886459    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:04:43.980663 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:04:43.980629    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-050e1b21-gw-openshift-default-f84bf8855-95cxj"]
Jul 14 12:04:43.981140 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:04:43.981121    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2dfc09e4-5fe7-4f67-9860-b8ede93c6eaa" containerName="maas-api"
Jul 14 12:04:43.981187 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:04:43.981145    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="2dfc09e4-5fe7-4f67-9860-b8ede93c6eaa" containerName="maas-api"
Jul 14 12:04:43.981271 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:04:43.981258    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="2dfc09e4-5fe7-4f67-9860-b8ede93c6eaa" containerName="maas-api"
Jul 14 12:04:43.984490 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:04:43.984472    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-050e1b21-gw-openshift-default-f84bf8855-95cxj"
Jul 14 12:04:43.988433 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:04:43.988198    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-050e1b21-gw-openshift-default-dockercfg-fmgvb\""
Jul 14 12:04:43.997108 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:04:43.997085    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-050e1b21-gw-openshift-default-f84bf8855-95cxj"]
Jul 14 12:04:44.023437 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:04:44.023407    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:04:44.066507 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:04:44.066408    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/114ad6f5-d49d-4429-923f-3b77bbf78b6a-istio-data\") pod \"e2e-migrate-050e1b21-gw-openshift-default-f84bf8855-95cxj\" (UID: \"114ad6f5-d49d-4429-923f-3b77bbf78b6a\") " pod="openshift-ingress/e2e-migrate-050e1b21-gw-openshift-default-f84bf8855-95cxj"
Jul 14 12:04:44.066507 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:04:44.066450    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/114ad6f5-d49d-4429-923f-3b77bbf78b6a-workload-certs\") pod \"e2e-migrate-050e1b21-gw-openshift-default-f84bf8855-95cxj\" (UID: \"114ad6f5-d49d-4429-923f-3b77bbf78b6a\") " pod="openshift-ingress/e2e-migrate-050e1b21-gw-openshift-default-f84bf8855-95cxj"
Jul 14 12:04:44.066507 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:04:44.066480    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfvnr\" (UniqueName: \"kubernetes.io/projected/114ad6f5-d49d-4429-923f-3b77bbf78b6a-kube-api-access-lfvnr\") pod \"e2e-migrate-050e1b21-gw-openshift-default-f84bf8855-95cxj\" (UID: \"114ad6f5-d49d-4429-923f-3b77bbf78b6a\") " pod="openshift-ingress/e2e-migrate-050e1b21-gw-openshift-default-f84bf8855-95cxj"
Jul 14 12:04:44.066507 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:04:44.066506    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/114ad6f5-d49d-4429-923f-3b77bbf78b6a-istio-envoy\") pod \"e2e-migrate-050e1b21-gw-openshift-default-f84bf8855-95cxj\" (UID: \"114ad6f5-d49d-4429-923f-3b77bbf78b6a\") " pod="openshift-ingress/e2e-migrate-050e1b21-gw-openshift-default-f84bf8855-95cxj"
Jul 14 12:04:44.066902 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:04:44.066537    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/114ad6f5-d49d-4429-923f-3b77bbf78b6a-istio-podinfo\") pod \"e2e-migrate-050e1b21-gw-openshift-default-f84bf8855-95cxj\" (UID: \"114ad6f5-d49d-4429-923f-3b77bbf78b6a\") " pod="openshift-ingress/e2e-migrate-050e1b21-gw-openshift-default-f84bf8855-95cxj"
Jul 14 12:04:44.066902 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:04:44.066554    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/114ad6f5-d49d-4429-923f-3b77bbf78b6a-istiod-ca-cert\") pod \"e2e-migrate-050e1b21-gw-openshift-default-f84bf8855-95cxj\" (UID: \"114ad6f5-d49d-4429-923f-3b77bbf78b6a\") " pod="openshift-ingress/e2e-migrate-050e1b21-gw-openshift-default-f84bf8855-95cxj"
Jul 14 12:04:44.066902 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:04:44.066642    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/114ad6f5-d49d-4429-923f-3b77bbf78b6a-istio-token\") pod \"e2e-migrate-050e1b21-gw-openshift-default-f84bf8855-95cxj\" (UID: \"114ad6f5-d49d-4429-923f-3b77bbf78b6a\") " pod="openshift-ingress/e2e-migrate-050e1b21-gw-openshift-default-f84bf8855-95cxj"
Jul 14 12:04:44.066902 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:04:44.066673    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/114ad6f5-d49d-4429-923f-3b77bbf78b6a-workload-socket\") pod \"e2e-migrate-050e1b21-gw-openshift-default-f84bf8855-95cxj\" (UID: \"114ad6f5-d49d-4429-923f-3b77bbf78b6a\") " pod="openshift-ingress/e2e-migrate-050e1b21-gw-openshift-default-f84bf8855-95cxj"
Jul 14 12:04:44.066902 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:04:44.066693    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/114ad6f5-d49d-4429-923f-3b77bbf78b6a-credential-socket\") pod \"e2e-migrate-050e1b21-gw-openshift-default-f84bf8855-95cxj\" (UID: \"114ad6f5-d49d-4429-923f-3b77bbf78b6a\") " pod="openshift-ingress/e2e-migrate-050e1b21-gw-openshift-default-f84bf8855-95cxj"
Jul 14 12:04:44.168120 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:04:44.168078    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfvnr\" (UniqueName: \"kubernetes.io/projected/114ad6f5-d49d-4429-923f-3b77bbf78b6a-kube-api-access-lfvnr\") pod \"e2e-migrate-050e1b21-gw-openshift-default-f84bf8855-95cxj\" (UID: \"114ad6f5-d49d-4429-923f-3b77bbf78b6a\") " pod="openshift-ingress/e2e-migrate-050e1b21-gw-openshift-default-f84bf8855-95cxj"
Jul 14 12:04:44.168120 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:04:44.168124    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/114ad6f5-d49d-4429-923f-3b77bbf78b6a-istio-envoy\") pod \"e2e-migrate-050e1b21-gw-openshift-default-f84bf8855-95cxj\" (UID: \"114ad6f5-d49d-4429-923f-3b77bbf78b6a\") " pod="openshift-ingress/e2e-migrate-050e1b21-gw-openshift-default-f84bf8855-95cxj"
Jul 14 12:04:44.168385 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:04:44.168262    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/114ad6f5-d49d-4429-923f-3b77bbf78b6a-istio-podinfo\") pod \"e2e-migrate-050e1b21-gw-openshift-default-f84bf8855-95cxj\" (UID: \"114ad6f5-d49d-4429-923f-3b77bbf78b6a\") " pod="openshift-ingress/e2e-migrate-050e1b21-gw-openshift-default-f84bf8855-95cxj"
Jul 14 12:04:44.168385 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:04:44.168306    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/114ad6f5-d49d-4429-923f-3b77bbf78b6a-istiod-ca-cert\") pod \"e2e-migrate-050e1b21-gw-openshift-default-f84bf8855-95cxj\" (UID: \"114ad6f5-d49d-4429-923f-3b77bbf78b6a\") " pod="openshift-ingress/e2e-migrate-050e1b21-gw-openshift-default-f84bf8855-95cxj"
Jul 14 12:04:44.168385 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:04:44.168355    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/114ad6f5-d49d-4429-923f-3b77bbf78b6a-istio-token\") pod \"e2e-migrate-050e1b21-gw-openshift-default-f84bf8855-95cxj\" (UID: \"114ad6f5-d49d-4429-923f-3b77bbf78b6a\") " pod="openshift-ingress/e2e-migrate-050e1b21-gw-openshift-default-f84bf8855-95cxj"
Jul 14 12:04:44.168540 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:04:44.168389    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/114ad6f5-d49d-4429-923f-3b77bbf78b6a-workload-socket\") pod \"e2e-migrate-050e1b21-gw-openshift-default-f84bf8855-95cxj\" (UID: \"114ad6f5-d49d-4429-923f-3b77bbf78b6a\") " pod="openshift-ingress/e2e-migrate-050e1b21-gw-openshift-default-f84bf8855-95cxj"
Jul 14 12:04:44.168540 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:04:44.168418    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/114ad6f5-d49d-4429-923f-3b77bbf78b6a-credential-socket\") pod \"e2e-migrate-050e1b21-gw-openshift-default-f84bf8855-95cxj\" (UID: \"114ad6f5-d49d-4429-923f-3b77bbf78b6a\") " pod="openshift-ingress/e2e-migrate-050e1b21-gw-openshift-default-f84bf8855-95cxj"
Jul 14 12:04:44.168540 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:04:44.168483    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/114ad6f5-d49d-4429-923f-3b77bbf78b6a-istio-data\") pod \"e2e-migrate-050e1b21-gw-openshift-default-f84bf8855-95cxj\" (UID: \"114ad6f5-d49d-4429-923f-3b77bbf78b6a\") " pod="openshift-ingress/e2e-migrate-050e1b21-gw-openshift-default-f84bf8855-95cxj"
Jul 14 12:04:44.168540 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:04:44.168513    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/114ad6f5-d49d-4429-923f-3b77bbf78b6a-workload-certs\") pod \"e2e-migrate-050e1b21-gw-openshift-default-f84bf8855-95cxj\" (UID: \"114ad6f5-d49d-4429-923f-3b77bbf78b6a\") " pod="openshift-ingress/e2e-migrate-050e1b21-gw-openshift-default-f84bf8855-95cxj"
Jul 14 12:04:44.168807 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:04:44.168746    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/114ad6f5-d49d-4429-923f-3b77bbf78b6a-workload-socket\") pod \"e2e-migrate-050e1b21-gw-openshift-default-f84bf8855-95cxj\" (UID: \"114ad6f5-d49d-4429-923f-3b77bbf78b6a\") " pod="openshift-ingress/e2e-migrate-050e1b21-gw-openshift-default-f84bf8855-95cxj"
Jul 14 12:04:44.168859 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:04:44.168837    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/114ad6f5-d49d-4429-923f-3b77bbf78b6a-credential-socket\") pod \"e2e-migrate-050e1b21-gw-openshift-default-f84bf8855-95cxj\" (UID: \"114ad6f5-d49d-4429-923f-3b77bbf78b6a\") " pod="openshift-ingress/e2e-migrate-050e1b21-gw-openshift-default-f84bf8855-95cxj"
Jul 14 12:04:44.168930 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:04:44.168905    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/114ad6f5-d49d-4429-923f-3b77bbf78b6a-istio-data\") pod \"e2e-migrate-050e1b21-gw-openshift-default-f84bf8855-95cxj\" (UID: \"114ad6f5-d49d-4429-923f-3b77bbf78b6a\") " pod="openshift-ingress/e2e-migrate-050e1b21-gw-openshift-default-f84bf8855-95cxj"
Jul 14 12:04:44.169015 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:04:44.168998    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/114ad6f5-d49d-4429-923f-3b77bbf78b6a-workload-certs\") pod \"e2e-migrate-050e1b21-gw-openshift-default-f84bf8855-95cxj\" (UID: \"114ad6f5-d49d-4429-923f-3b77bbf78b6a\") " pod="openshift-ingress/e2e-migrate-050e1b21-gw-openshift-default-f84bf8855-95cxj"
Jul 14 12:04:44.169215 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:04:44.169194    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/114ad6f5-d49d-4429-923f-3b77bbf78b6a-istiod-ca-cert\") pod \"e2e-migrate-050e1b21-gw-openshift-default-f84bf8855-95cxj\" (UID: \"114ad6f5-d49d-4429-923f-3b77bbf78b6a\") " pod="openshift-ingress/e2e-migrate-050e1b21-gw-openshift-default-f84bf8855-95cxj"
Jul 14 12:04:44.170423 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:04:44.170395    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/114ad6f5-d49d-4429-923f-3b77bbf78b6a-istio-envoy\") pod \"e2e-migrate-050e1b21-gw-openshift-default-f84bf8855-95cxj\" (UID: \"114ad6f5-d49d-4429-923f-3b77bbf78b6a\") " pod="openshift-ingress/e2e-migrate-050e1b21-gw-openshift-default-f84bf8855-95cxj"
Jul 14 12:04:44.170783 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:04:44.170766    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/114ad6f5-d49d-4429-923f-3b77bbf78b6a-istio-podinfo\") pod \"e2e-migrate-050e1b21-gw-openshift-default-f84bf8855-95cxj\" (UID: \"114ad6f5-d49d-4429-923f-3b77bbf78b6a\") " pod="openshift-ingress/e2e-migrate-050e1b21-gw-openshift-default-f84bf8855-95cxj"
Jul 14 12:04:44.181301 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:04:44.181274    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/114ad6f5-d49d-4429-923f-3b77bbf78b6a-istio-token\") pod \"e2e-migrate-050e1b21-gw-openshift-default-f84bf8855-95cxj\" (UID: \"114ad6f5-d49d-4429-923f-3b77bbf78b6a\") " pod="openshift-ingress/e2e-migrate-050e1b21-gw-openshift-default-f84bf8855-95cxj"
Jul 14 12:04:44.181568 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:04:44.181545    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfvnr\" (UniqueName: \"kubernetes.io/projected/114ad6f5-d49d-4429-923f-3b77bbf78b6a-kube-api-access-lfvnr\") pod \"e2e-migrate-050e1b21-gw-openshift-default-f84bf8855-95cxj\" (UID: \"114ad6f5-d49d-4429-923f-3b77bbf78b6a\") " pod="openshift-ingress/e2e-migrate-050e1b21-gw-openshift-default-f84bf8855-95cxj"
Jul 14 12:04:44.295768 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:04:44.295734    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-050e1b21-gw-openshift-default-f84bf8855-95cxj"
Jul 14 12:04:44.433643 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:04:44.433609    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-050e1b21-gw-openshift-default-f84bf8855-95cxj"]
Jul 14 12:04:44.437140 ip-10-0-136-173 kubenswrapper[2551]: W0714 12:04:44.437116    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod114ad6f5_d49d_4429_923f_3b77bbf78b6a.slice/crio-7920746963cceb9a8ea4a527b510a7e643b6d40e9a6f094bd4836bd414d5d80e WatchSource:0}: Error finding container 7920746963cceb9a8ea4a527b510a7e643b6d40e9a6f094bd4836bd414d5d80e: Status 404 returned error can't find the container with id 7920746963cceb9a8ea4a527b510a7e643b6d40e9a6f094bd4836bd414d5d80e
Jul 14 12:04:44.439147 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:04:44.439114    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 14 12:04:44.439232 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:04:44.439179    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 14 12:04:44.439232 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:04:44.439208    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 14 12:04:44.942651 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:04:44.942614    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:04:45.315640 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:04:45.315524    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:04:45.373201 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:04:45.373168    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-050e1b21-787c975778-rtw4v"]
Jul 14 12:04:45.376766 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:04:45.376742    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-050e1b21-787c975778-rtw4v"
Jul 14 12:04:45.379430 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:04:45.379409    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-050e1b21-serving-cert\""
Jul 14 12:04:45.379566 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:04:45.379439    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 14 12:04:45.379696 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:04:45.379672    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 14 12:04:45.381750 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:04:45.381731    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-62k4x\""
Jul 14 12:04:45.390555 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:04:45.390534    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-050e1b21-787c975778-rtw4v"]
Jul 14 12:04:45.408164 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:04:45.408139    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-050e1b21-gw-openshift-default-f84bf8855-95cxj" event={"ID":"114ad6f5-d49d-4429-923f-3b77bbf78b6a","Type":"ContainerStarted","Data":"6100e6d2d9c9407295c4b291da0fdc15dbe19b7099f1f7097dc5bb156131b41a"}
Jul 14 12:04:45.408282 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:04:45.408183    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-050e1b21-gw-openshift-default-f84bf8855-95cxj" event={"ID":"114ad6f5-d49d-4429-923f-3b77bbf78b6a","Type":"ContainerStarted","Data":"7920746963cceb9a8ea4a527b510a7e643b6d40e9a6f094bd4836bd414d5d80e"}
Jul 14 12:04:45.431227 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:04:45.431104    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-050e1b21-gw-openshift-default-f84bf8855-95cxj" podStartSLOduration=2.431085232 podStartE2EDuration="2.431085232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 12:04:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 12:04:45.430482409 +0000 UTC m=+1636.662291706" watchObservedRunningTime="2026-07-14 12:04:45.431085232 +0000 UTC m=+1636.662894528"
Jul 14 12:04:45.435894 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:04:45.435868    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:04:45.480224 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:04:45.480188    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd57e1a0-e2f8-416a-992e-72baa6e50972-maas-api-tls\") pod \"maas-api-e2e-migrate-050e1b21-787c975778-rtw4v\" (UID: \"fd57e1a0-e2f8-416a-992e-72baa6e50972\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-050e1b21-787c975778-rtw4v"
Jul 14 12:04:45.480367 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:04:45.480231    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqxjj\" (UniqueName: \"kubernetes.io/projected/fd57e1a0-e2f8-416a-992e-72baa6e50972-kube-api-access-mqxjj\") pod \"maas-api-e2e-migrate-050e1b21-787c975778-rtw4v\" (UID: \"fd57e1a0-e2f8-416a-992e-72baa6e50972\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-050e1b21-787c975778-rtw4v"
Jul 14 12:04:45.480527 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:04:45.480504    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd57e1a0-e2f8-416a-992e-72baa6e50972-openshift-service-ca\") pod \"maas-api-e2e-migrate-050e1b21-787c975778-rtw4v\" (UID: \"fd57e1a0-e2f8-416a-992e-72baa6e50972\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-050e1b21-787c975778-rtw4v"
Jul 14 12:04:45.581711 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:04:45.581619    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd57e1a0-e2f8-416a-992e-72baa6e50972-openshift-service-ca\") pod \"maas-api-e2e-migrate-050e1b21-787c975778-rtw4v\" (UID: \"fd57e1a0-e2f8-416a-992e-72baa6e50972\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-050e1b21-787c975778-rtw4v"
Jul 14 12:04:45.581711 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:04:45.581673    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd57e1a0-e2f8-416a-992e-72baa6e50972-maas-api-tls\") pod \"maas-api-e2e-migrate-050e1b21-787c975778-rtw4v\" (UID: \"fd57e1a0-e2f8-416a-992e-72baa6e50972\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-050e1b21-787c975778-rtw4v"
Jul 14 12:04:45.581711 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:04:45.581698    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqxjj\" (UniqueName: \"kubernetes.io/projected/fd57e1a0-e2f8-416a-992e-72baa6e50972-kube-api-access-mqxjj\") pod \"maas-api-e2e-migrate-050e1b21-787c975778-rtw4v\" (UID: \"fd57e1a0-e2f8-416a-992e-72baa6e50972\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-050e1b21-787c975778-rtw4v"
Jul 14 12:04:45.581995 ip-10-0-136-173 kubenswrapper[2551]: E0714 12:04:45.581831    2551 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-050e1b21-serving-cert: secret "maas-api-e2e-migrate-050e1b21-serving-cert" not found
Jul 14 12:04:45.581995 ip-10-0-136-173 kubenswrapper[2551]: E0714 12:04:45.581907    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd57e1a0-e2f8-416a-992e-72baa6e50972-maas-api-tls podName:fd57e1a0-e2f8-416a-992e-72baa6e50972 nodeName:}" failed. No retries permitted until 2026-07-14 12:04:46.081883324 +0000 UTC m=+1637.313692601 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fd57e1a0-e2f8-416a-992e-72baa6e50972-maas-api-tls") pod "maas-api-e2e-migrate-050e1b21-787c975778-rtw4v" (UID: "fd57e1a0-e2f8-416a-992e-72baa6e50972") : secret "maas-api-e2e-migrate-050e1b21-serving-cert" not found
Jul 14 12:04:45.582348 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:04:45.582328    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd57e1a0-e2f8-416a-992e-72baa6e50972-openshift-service-ca\") pod \"maas-api-e2e-migrate-050e1b21-787c975778-rtw4v\" (UID: \"fd57e1a0-e2f8-416a-992e-72baa6e50972\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-050e1b21-787c975778-rtw4v"
Jul 14 12:04:45.591062 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:04:45.591034    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqxjj\" (UniqueName: \"kubernetes.io/projected/fd57e1a0-e2f8-416a-992e-72baa6e50972-kube-api-access-mqxjj\") pod \"maas-api-e2e-migrate-050e1b21-787c975778-rtw4v\" (UID: \"fd57e1a0-e2f8-416a-992e-72baa6e50972\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-050e1b21-787c975778-rtw4v"
Jul 14 12:04:45.933339 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:04:45.933236    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:04:46.031375 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:04:46.031342    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:04:46.085462 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:04:46.085435    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd57e1a0-e2f8-416a-992e-72baa6e50972-maas-api-tls\") pod \"maas-api-e2e-migrate-050e1b21-787c975778-rtw4v\" (UID: \"fd57e1a0-e2f8-416a-992e-72baa6e50972\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-050e1b21-787c975778-rtw4v"
Jul 14 12:04:46.088035 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:04:46.088009    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd57e1a0-e2f8-416a-992e-72baa6e50972-maas-api-tls\") pod \"maas-api-e2e-migrate-050e1b21-787c975778-rtw4v\" (UID: \"fd57e1a0-e2f8-416a-992e-72baa6e50972\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-050e1b21-787c975778-rtw4v"
Jul 14 12:04:46.287479 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:04:46.287438    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-050e1b21-787c975778-rtw4v"
Jul 14 12:04:46.296735 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:04:46.296709    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-050e1b21-gw-openshift-default-f84bf8855-95cxj"
Jul 14 12:04:46.302007 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:04:46.301983    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-050e1b21-gw-openshift-default-f84bf8855-95cxj"
Jul 14 12:04:46.414030 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:04:46.413999    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-050e1b21-gw-openshift-default-f84bf8855-95cxj"
Jul 14 12:04:46.414910 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:04:46.414890    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-050e1b21-gw-openshift-default-f84bf8855-95cxj"
Jul 14 12:04:46.424980 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:04:46.424951    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-050e1b21-787c975778-rtw4v"]
Jul 14 12:04:46.427940 ip-10-0-136-173 kubenswrapper[2551]: W0714 12:04:46.427900    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd57e1a0_e2f8_416a_992e_72baa6e50972.slice/crio-41c05485afd2686985059ba821926077edf0cb3d26026d1c1de38eda56867595 WatchSource:0}: Error finding container 41c05485afd2686985059ba821926077edf0cb3d26026d1c1de38eda56867595: Status 404 returned error can't find the container with id 41c05485afd2686985059ba821926077edf0cb3d26026d1c1de38eda56867595
Jul 14 12:04:47.422423 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:04:47.422328    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-050e1b21-787c975778-rtw4v" event={"ID":"fd57e1a0-e2f8-416a-992e-72baa6e50972","Type":"ContainerStarted","Data":"f046dd2b91417a51b38aa26d74db5b5171ac4f61248cf7a46be3ced58e71cd29"}
Jul 14 12:04:47.422423 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:04:47.422394    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-050e1b21-787c975778-rtw4v" event={"ID":"fd57e1a0-e2f8-416a-992e-72baa6e50972","Type":"ContainerStarted","Data":"41c05485afd2686985059ba821926077edf0cb3d26026d1c1de38eda56867595"}
Jul 14 12:04:47.449880 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:04:47.449833    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-050e1b21-787c975778-rtw4v" podStartSLOduration=1.92147665 podStartE2EDuration="2.44981889s" totalImagesPullingTime="528.34224ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 12:04:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 12:04:47.448689312 +0000 UTC m=+1638.680498627" watchObservedRunningTime="2026-07-14 12:04:47.44981889 +0000 UTC m=+1638.681628185"
Jul 14 12:04:48.430270 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:04:48.430234    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-050e1b21-787c975778-rtw4v"
Jul 14 12:04:50.089796 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:04:50.089761    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:04:50.258441 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:04:50.258354    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:04:52.593542 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:04:52.593505    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 12:04:53.651374 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:04:53.651177    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:04:54.448873 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:04:54.448836    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-050e1b21-787c975778-rtw4v"
Jul 14 12:05:00.488858 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:00.488772    2551 generic.go:356] "Generic (PLEG): container finished" podID="114ad6f5-d49d-4429-923f-3b77bbf78b6a" containerID="6100e6d2d9c9407295c4b291da0fdc15dbe19b7099f1f7097dc5bb156131b41a" exitCode=0
Jul 14 12:05:00.488858 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:00.488843    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-050e1b21-gw-openshift-default-f84bf8855-95cxj" event={"ID":"114ad6f5-d49d-4429-923f-3b77bbf78b6a","Type":"ContainerDied","Data":"6100e6d2d9c9407295c4b291da0fdc15dbe19b7099f1f7097dc5bb156131b41a"}
Jul 14 12:05:00.489277 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:00.489252    2551 scope.go:117] "RemoveContainer" containerID="6100e6d2d9c9407295c4b291da0fdc15dbe19b7099f1f7097dc5bb156131b41a"
Jul 14 12:05:00.490242 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:00.490214    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 14 12:05:00.490294 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:00.490279    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 14 12:05:00.490332 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:00.490314    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 14 12:05:01.414627 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:01.414596    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-050e1b21-gw-openshift-default-f84bf8855-95cxj"
Jul 14 12:05:01.494946 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:01.494909    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-050e1b21-gw-openshift-default-f84bf8855-95cxj" event={"ID":"114ad6f5-d49d-4429-923f-3b77bbf78b6a","Type":"ContainerStarted","Data":"b239bc9008b4e83d1ec090f27b05d817fe3708a9c72dee09e85273148ae372fa"}
Jul 14 12:05:02.296251 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:02.296219    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-050e1b21-gw-openshift-default-f84bf8855-95cxj"
Jul 14 12:05:02.297482 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:02.297459    2551 patch_prober.go:28] interesting pod/e2e-migrate-050e1b21-gw-openshift-default-f84bf8855-95cxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body=
Jul 14 12:05:02.297540 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:02.297510    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-050e1b21-gw-openshift-default-f84bf8855-95cxj" podUID="114ad6f5-d49d-4429-923f-3b77bbf78b6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused"
Jul 14 12:05:03.296743 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:03.296707    2551 patch_prober.go:28] interesting pod/e2e-migrate-050e1b21-gw-openshift-default-f84bf8855-95cxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body=
Jul 14 12:05:03.297144 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:03.296763    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-050e1b21-gw-openshift-default-f84bf8855-95cxj" podUID="114ad6f5-d49d-4429-923f-3b77bbf78b6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused"
Jul 14 12:05:04.296270 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:04.296231    2551 patch_prober.go:28] interesting pod/e2e-migrate-050e1b21-gw-openshift-default-f84bf8855-95cxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body=
Jul 14 12:05:04.296450 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:04.296287    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-050e1b21-gw-openshift-default-f84bf8855-95cxj" podUID="114ad6f5-d49d-4429-923f-3b77bbf78b6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused"
Jul 14 12:05:05.297044 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:05.297003    2551 patch_prober.go:28] interesting pod/e2e-migrate-050e1b21-gw-openshift-default-f84bf8855-95cxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body=
Jul 14 12:05:05.297442 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:05.297062    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-050e1b21-gw-openshift-default-f84bf8855-95cxj" podUID="114ad6f5-d49d-4429-923f-3b77bbf78b6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused"
Jul 14 12:05:05.425637 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:05.425600    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:05:06.300085 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:06.300055    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-050e1b21-gw-openshift-default-f84bf8855-95cxj"
Jul 14 12:05:06.300485 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:06.300318    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-050e1b21-gw-openshift-default-f84bf8855-95cxj"
Jul 14 12:05:06.300936 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:06.300919    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-050e1b21-gw-openshift-default-f84bf8855-95cxj"
Jul 14 12:05:49.903127 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:49.903090    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-050e1b21-gw-openshift-default-f84bf8855-95cxj"]
Jul 14 12:05:49.903574 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:49.903349    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-050e1b21-gw-openshift-default-f84bf8855-95cxj" podUID="114ad6f5-d49d-4429-923f-3b77bbf78b6a" containerName="istio-proxy" containerID="cri-o://b239bc9008b4e83d1ec090f27b05d817fe3708a9c72dee09e85273148ae372fa" gracePeriod=30
Jul 14 12:05:49.932150 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:49.932123    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:05:50.147266 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:50.147232    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:05:50.281789 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:50.281760    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-050e1b21-787c975778-rtw4v"]
Jul 14 12:05:50.282025 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:50.281997    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-050e1b21-787c975778-rtw4v" podUID="fd57e1a0-e2f8-416a-992e-72baa6e50972" containerName="maas-api" containerID="cri-o://f046dd2b91417a51b38aa26d74db5b5171ac4f61248cf7a46be3ced58e71cd29" gracePeriod=30
Jul 14 12:05:50.311907 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:50.311874    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:05:50.537928 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:50.537868    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-050e1b21-787c975778-rtw4v"
Jul 14 12:05:50.660078 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:50.660044    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mqxjj\" (UniqueName: \"kubernetes.io/projected/fd57e1a0-e2f8-416a-992e-72baa6e50972-kube-api-access-mqxjj\") pod \"fd57e1a0-e2f8-416a-992e-72baa6e50972\" (UID: \"fd57e1a0-e2f8-416a-992e-72baa6e50972\") "
Jul 14 12:05:50.660257 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:50.660118    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd57e1a0-e2f8-416a-992e-72baa6e50972-openshift-service-ca\") pod \"fd57e1a0-e2f8-416a-992e-72baa6e50972\" (UID: \"fd57e1a0-e2f8-416a-992e-72baa6e50972\") "
Jul 14 12:05:50.660257 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:50.660152    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd57e1a0-e2f8-416a-992e-72baa6e50972-maas-api-tls\") pod \"fd57e1a0-e2f8-416a-992e-72baa6e50972\" (UID: \"fd57e1a0-e2f8-416a-992e-72baa6e50972\") "
Jul 14 12:05:50.660471 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:50.660441    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd57e1a0-e2f8-416a-992e-72baa6e50972-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fd57e1a0-e2f8-416a-992e-72baa6e50972" (UID: "fd57e1a0-e2f8-416a-992e-72baa6e50972"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 12:05:50.662216 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:50.662185    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd57e1a0-e2f8-416a-992e-72baa6e50972-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fd57e1a0-e2f8-416a-992e-72baa6e50972" (UID: "fd57e1a0-e2f8-416a-992e-72baa6e50972"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 12:05:50.662318 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:50.662227    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd57e1a0-e2f8-416a-992e-72baa6e50972-kube-api-access-mqxjj" (OuterVolumeSpecName: "kube-api-access-mqxjj") pod "fd57e1a0-e2f8-416a-992e-72baa6e50972" (UID: "fd57e1a0-e2f8-416a-992e-72baa6e50972"). InnerVolumeSpecName "kube-api-access-mqxjj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 12:05:50.721448 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:50.721421    2551 generic.go:356] "Generic (PLEG): container finished" podID="fd57e1a0-e2f8-416a-992e-72baa6e50972" containerID="f046dd2b91417a51b38aa26d74db5b5171ac4f61248cf7a46be3ced58e71cd29" exitCode=0
Jul 14 12:05:50.721559 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:50.721458    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-050e1b21-787c975778-rtw4v"
Jul 14 12:05:50.721559 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:50.721470    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-050e1b21-787c975778-rtw4v" event={"ID":"fd57e1a0-e2f8-416a-992e-72baa6e50972","Type":"ContainerDied","Data":"f046dd2b91417a51b38aa26d74db5b5171ac4f61248cf7a46be3ced58e71cd29"}
Jul 14 12:05:50.721559 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:50.721506    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-050e1b21-787c975778-rtw4v" event={"ID":"fd57e1a0-e2f8-416a-992e-72baa6e50972","Type":"ContainerDied","Data":"41c05485afd2686985059ba821926077edf0cb3d26026d1c1de38eda56867595"}
Jul 14 12:05:50.721559 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:50.721527    2551 scope.go:117] "RemoveContainer" containerID="f046dd2b91417a51b38aa26d74db5b5171ac4f61248cf7a46be3ced58e71cd29"
Jul 14 12:05:50.731525 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:50.731507    2551 scope.go:117] "RemoveContainer" containerID="f046dd2b91417a51b38aa26d74db5b5171ac4f61248cf7a46be3ced58e71cd29"
Jul 14 12:05:50.731801 ip-10-0-136-173 kubenswrapper[2551]: E0714 12:05:50.731778    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f046dd2b91417a51b38aa26d74db5b5171ac4f61248cf7a46be3ced58e71cd29\": container with ID starting with f046dd2b91417a51b38aa26d74db5b5171ac4f61248cf7a46be3ced58e71cd29 not found: ID does not exist" containerID="f046dd2b91417a51b38aa26d74db5b5171ac4f61248cf7a46be3ced58e71cd29"
Jul 14 12:05:50.731888 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:50.731816    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f046dd2b91417a51b38aa26d74db5b5171ac4f61248cf7a46be3ced58e71cd29"} err="failed to get container status \"f046dd2b91417a51b38aa26d74db5b5171ac4f61248cf7a46be3ced58e71cd29\": rpc error: code = NotFound desc = could not find container \"f046dd2b91417a51b38aa26d74db5b5171ac4f61248cf7a46be3ced58e71cd29\": container with ID starting with f046dd2b91417a51b38aa26d74db5b5171ac4f61248cf7a46be3ced58e71cd29 not found: ID does not exist"
Jul 14 12:05:50.746917 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:50.746897    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-050e1b21-787c975778-rtw4v"]
Jul 14 12:05:50.749826 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:50.749807    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-050e1b21-787c975778-rtw4v"]
Jul 14 12:05:50.761544 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:50.761524    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqxjj\" (UniqueName: \"kubernetes.io/projected/fd57e1a0-e2f8-416a-992e-72baa6e50972-kube-api-access-mqxjj\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 12:05:50.761544 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:50.761542    2551 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd57e1a0-e2f8-416a-992e-72baa6e50972-openshift-service-ca\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 12:05:50.761668 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:50.761554    2551 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd57e1a0-e2f8-416a-992e-72baa6e50972-maas-api-tls\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 12:05:51.191066 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:51.191024    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd57e1a0-e2f8-416a-992e-72baa6e50972" path="/var/lib/kubelet/pods/fd57e1a0-e2f8-416a-992e-72baa6e50972/volumes"
Jul 14 12:05:51.303266 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:51.303228    2551 patch_prober.go:28] interesting pod/e2e-migrate-050e1b21-gw-openshift-default-f84bf8855-95cxj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 14 12:05:51.303419 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:51.303295    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-050e1b21-gw-openshift-default-f84bf8855-95cxj" podUID="114ad6f5-d49d-4429-923f-3b77bbf78b6a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 14 12:05:52.894691 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:52.894655    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 12:05:55.167265 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:55.167202    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-050e1b21-gw-openshift-default-f84bf8855-95cxj"
Jul 14 12:05:55.303084 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:55.303043    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/114ad6f5-d49d-4429-923f-3b77bbf78b6a-workload-certs\") pod \"114ad6f5-d49d-4429-923f-3b77bbf78b6a\" (UID: \"114ad6f5-d49d-4429-923f-3b77bbf78b6a\") "
Jul 14 12:05:55.303242 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:55.303091    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lfvnr\" (UniqueName: \"kubernetes.io/projected/114ad6f5-d49d-4429-923f-3b77bbf78b6a-kube-api-access-lfvnr\") pod \"114ad6f5-d49d-4429-923f-3b77bbf78b6a\" (UID: \"114ad6f5-d49d-4429-923f-3b77bbf78b6a\") "
Jul 14 12:05:55.303242 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:55.303119    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/114ad6f5-d49d-4429-923f-3b77bbf78b6a-istio-podinfo\") pod \"114ad6f5-d49d-4429-923f-3b77bbf78b6a\" (UID: \"114ad6f5-d49d-4429-923f-3b77bbf78b6a\") "
Jul 14 12:05:55.303242 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:55.303218    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/114ad6f5-d49d-4429-923f-3b77bbf78b6a-istio-data\") pod \"114ad6f5-d49d-4429-923f-3b77bbf78b6a\" (UID: \"114ad6f5-d49d-4429-923f-3b77bbf78b6a\") "
Jul 14 12:05:55.303242 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:55.303239    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/114ad6f5-d49d-4429-923f-3b77bbf78b6a-workload-socket\") pod \"114ad6f5-d49d-4429-923f-3b77bbf78b6a\" (UID: \"114ad6f5-d49d-4429-923f-3b77bbf78b6a\") "
Jul 14 12:05:55.303430 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:55.303259    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/114ad6f5-d49d-4429-923f-3b77bbf78b6a-credential-socket\") pod \"114ad6f5-d49d-4429-923f-3b77bbf78b6a\" (UID: \"114ad6f5-d49d-4429-923f-3b77bbf78b6a\") "
Jul 14 12:05:55.303430 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:55.303280    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/114ad6f5-d49d-4429-923f-3b77bbf78b6a-istio-token\") pod \"114ad6f5-d49d-4429-923f-3b77bbf78b6a\" (UID: \"114ad6f5-d49d-4429-923f-3b77bbf78b6a\") "
Jul 14 12:05:55.303430 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:55.303376    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/114ad6f5-d49d-4429-923f-3b77bbf78b6a-istiod-ca-cert\") pod \"114ad6f5-d49d-4429-923f-3b77bbf78b6a\" (UID: \"114ad6f5-d49d-4429-923f-3b77bbf78b6a\") "
Jul 14 12:05:55.303430 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:55.303379    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/114ad6f5-d49d-4429-923f-3b77bbf78b6a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "114ad6f5-d49d-4429-923f-3b77bbf78b6a" (UID: "114ad6f5-d49d-4429-923f-3b77bbf78b6a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 12:05:55.303430 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:55.303407    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/114ad6f5-d49d-4429-923f-3b77bbf78b6a-istio-envoy\") pod \"114ad6f5-d49d-4429-923f-3b77bbf78b6a\" (UID: \"114ad6f5-d49d-4429-923f-3b77bbf78b6a\") "
Jul 14 12:05:55.303704 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:55.303497    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/114ad6f5-d49d-4429-923f-3b77bbf78b6a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "114ad6f5-d49d-4429-923f-3b77bbf78b6a" (UID: "114ad6f5-d49d-4429-923f-3b77bbf78b6a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 12:05:55.303704 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:55.303600    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/114ad6f5-d49d-4429-923f-3b77bbf78b6a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "114ad6f5-d49d-4429-923f-3b77bbf78b6a" (UID: "114ad6f5-d49d-4429-923f-3b77bbf78b6a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 12:05:55.303801 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:55.303731    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/114ad6f5-d49d-4429-923f-3b77bbf78b6a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "114ad6f5-d49d-4429-923f-3b77bbf78b6a" (UID: "114ad6f5-d49d-4429-923f-3b77bbf78b6a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 12:05:55.303872 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:55.303852    2551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/114ad6f5-d49d-4429-923f-3b77bbf78b6a-credential-socket\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 12:05:55.303928 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:55.303879    2551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/114ad6f5-d49d-4429-923f-3b77bbf78b6a-istiod-ca-cert\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 12:05:55.303928 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:55.303896    2551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/114ad6f5-d49d-4429-923f-3b77bbf78b6a-workload-certs\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 12:05:55.303928 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:55.303912    2551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/114ad6f5-d49d-4429-923f-3b77bbf78b6a-workload-socket\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 12:05:55.304126 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:55.304096    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/114ad6f5-d49d-4429-923f-3b77bbf78b6a-istio-data" (OuterVolumeSpecName: "istio-data") pod "114ad6f5-d49d-4429-923f-3b77bbf78b6a" (UID: "114ad6f5-d49d-4429-923f-3b77bbf78b6a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 12:05:55.305469 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:55.305438    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/114ad6f5-d49d-4429-923f-3b77bbf78b6a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "114ad6f5-d49d-4429-923f-3b77bbf78b6a" (UID: "114ad6f5-d49d-4429-923f-3b77bbf78b6a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 12:05:55.305825 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:55.305795    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/114ad6f5-d49d-4429-923f-3b77bbf78b6a-kube-api-access-lfvnr" (OuterVolumeSpecName: "kube-api-access-lfvnr") pod "114ad6f5-d49d-4429-923f-3b77bbf78b6a" (UID: "114ad6f5-d49d-4429-923f-3b77bbf78b6a"). InnerVolumeSpecName "kube-api-access-lfvnr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 12:05:55.305825 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:55.305796    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/114ad6f5-d49d-4429-923f-3b77bbf78b6a-istio-token" (OuterVolumeSpecName: "istio-token") pod "114ad6f5-d49d-4429-923f-3b77bbf78b6a" (UID: "114ad6f5-d49d-4429-923f-3b77bbf78b6a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 12:05:55.305952 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:55.305920    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/114ad6f5-d49d-4429-923f-3b77bbf78b6a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "114ad6f5-d49d-4429-923f-3b77bbf78b6a" (UID: "114ad6f5-d49d-4429-923f-3b77bbf78b6a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 14 12:05:55.404871 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:55.404831    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfvnr\" (UniqueName: \"kubernetes.io/projected/114ad6f5-d49d-4429-923f-3b77bbf78b6a-kube-api-access-lfvnr\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 12:05:55.404871 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:55.404863    2551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/114ad6f5-d49d-4429-923f-3b77bbf78b6a-istio-podinfo\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 12:05:55.404871 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:55.404873    2551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/114ad6f5-d49d-4429-923f-3b77bbf78b6a-istio-data\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 12:05:55.404871 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:55.404882    2551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/114ad6f5-d49d-4429-923f-3b77bbf78b6a-istio-token\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 12:05:55.405216 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:55.404890    2551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/114ad6f5-d49d-4429-923f-3b77bbf78b6a-istio-envoy\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 12:05:55.747248 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:55.747213    2551 generic.go:356] "Generic (PLEG): container finished" podID="114ad6f5-d49d-4429-923f-3b77bbf78b6a" containerID="b239bc9008b4e83d1ec090f27b05d817fe3708a9c72dee09e85273148ae372fa" exitCode=0
Jul 14 12:05:55.747421 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:55.747284    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-050e1b21-gw-openshift-default-f84bf8855-95cxj"
Jul 14 12:05:55.747421 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:55.747289    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-050e1b21-gw-openshift-default-f84bf8855-95cxj" event={"ID":"114ad6f5-d49d-4429-923f-3b77bbf78b6a","Type":"ContainerDied","Data":"b239bc9008b4e83d1ec090f27b05d817fe3708a9c72dee09e85273148ae372fa"}
Jul 14 12:05:55.747421 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:55.747342    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-050e1b21-gw-openshift-default-f84bf8855-95cxj" event={"ID":"114ad6f5-d49d-4429-923f-3b77bbf78b6a","Type":"ContainerDied","Data":"7920746963cceb9a8ea4a527b510a7e643b6d40e9a6f094bd4836bd414d5d80e"}
Jul 14 12:05:55.747421 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:55.747365    2551 scope.go:117] "RemoveContainer" containerID="b239bc9008b4e83d1ec090f27b05d817fe3708a9c72dee09e85273148ae372fa"
Jul 14 12:05:55.756792 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:55.756773    2551 scope.go:117] "RemoveContainer" containerID="6100e6d2d9c9407295c4b291da0fdc15dbe19b7099f1f7097dc5bb156131b41a"
Jul 14 12:05:55.764348 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:55.764329    2551 scope.go:117] "RemoveContainer" containerID="b239bc9008b4e83d1ec090f27b05d817fe3708a9c72dee09e85273148ae372fa"
Jul 14 12:05:55.764624 ip-10-0-136-173 kubenswrapper[2551]: E0714 12:05:55.764601    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b239bc9008b4e83d1ec090f27b05d817fe3708a9c72dee09e85273148ae372fa\": container with ID starting with b239bc9008b4e83d1ec090f27b05d817fe3708a9c72dee09e85273148ae372fa not found: ID does not exist" containerID="b239bc9008b4e83d1ec090f27b05d817fe3708a9c72dee09e85273148ae372fa"
Jul 14 12:05:55.764711 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:55.764635    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b239bc9008b4e83d1ec090f27b05d817fe3708a9c72dee09e85273148ae372fa"} err="failed to get container status \"b239bc9008b4e83d1ec090f27b05d817fe3708a9c72dee09e85273148ae372fa\": rpc error: code = NotFound desc = could not find container \"b239bc9008b4e83d1ec090f27b05d817fe3708a9c72dee09e85273148ae372fa\": container with ID starting with b239bc9008b4e83d1ec090f27b05d817fe3708a9c72dee09e85273148ae372fa not found: ID does not exist"
Jul 14 12:05:55.764711 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:55.764660    2551 scope.go:117] "RemoveContainer" containerID="6100e6d2d9c9407295c4b291da0fdc15dbe19b7099f1f7097dc5bb156131b41a"
Jul 14 12:05:55.764924 ip-10-0-136-173 kubenswrapper[2551]: E0714 12:05:55.764905    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6100e6d2d9c9407295c4b291da0fdc15dbe19b7099f1f7097dc5bb156131b41a\": container with ID starting with 6100e6d2d9c9407295c4b291da0fdc15dbe19b7099f1f7097dc5bb156131b41a not found: ID does not exist" containerID="6100e6d2d9c9407295c4b291da0fdc15dbe19b7099f1f7097dc5bb156131b41a"
Jul 14 12:05:55.764982 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:55.764932    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6100e6d2d9c9407295c4b291da0fdc15dbe19b7099f1f7097dc5bb156131b41a"} err="failed to get container status \"6100e6d2d9c9407295c4b291da0fdc15dbe19b7099f1f7097dc5bb156131b41a\": rpc error: code = NotFound desc = could not find container \"6100e6d2d9c9407295c4b291da0fdc15dbe19b7099f1f7097dc5bb156131b41a\": container with ID starting with 6100e6d2d9c9407295c4b291da0fdc15dbe19b7099f1f7097dc5bb156131b41a not found: ID does not exist"
Jul 14 12:05:55.771682 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:55.771663    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-050e1b21-gw-openshift-default-f84bf8855-95cxj"]
Jul 14 12:05:55.777232 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:55.777207    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-050e1b21-gw-openshift-default-f84bf8855-95cxj"]
Jul 14 12:05:56.536135 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:56.536103    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:05:56.636959 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:56.636905    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:05:56.780740 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:56.780710    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:05:56.844119 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:56.844045    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:05:57.191246 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:57.191218    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="114ad6f5-d49d-4429-923f-3b77bbf78b6a" path="/var/lib/kubelet/pods/114ad6f5-d49d-4429-923f-3b77bbf78b6a/volumes"
Jul 14 12:05:57.957257 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:57.957211    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:05:58.039138 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:58.039107    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0135d283-677488f497-xmv59"]
Jul 14 12:05:58.039468 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:58.039452    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd57e1a0-e2f8-416a-992e-72baa6e50972" containerName="maas-api"
Jul 14 12:05:58.039468 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:58.039468    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd57e1a0-e2f8-416a-992e-72baa6e50972" containerName="maas-api"
Jul 14 12:05:58.039625 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:58.039482    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="114ad6f5-d49d-4429-923f-3b77bbf78b6a" containerName="istio-proxy"
Jul 14 12:05:58.039625 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:58.039488    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="114ad6f5-d49d-4429-923f-3b77bbf78b6a" containerName="istio-proxy"
Jul 14 12:05:58.039625 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:58.039501    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="114ad6f5-d49d-4429-923f-3b77bbf78b6a" containerName="istio-proxy"
Jul 14 12:05:58.039625 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:58.039510    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="114ad6f5-d49d-4429-923f-3b77bbf78b6a" containerName="istio-proxy"
Jul 14 12:05:58.039625 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:58.039620    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="114ad6f5-d49d-4429-923f-3b77bbf78b6a" containerName="istio-proxy"
Jul 14 12:05:58.039792 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:58.039635    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd57e1a0-e2f8-416a-992e-72baa6e50972" containerName="maas-api"
Jul 14 12:05:58.039792 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:58.039646    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="114ad6f5-d49d-4429-923f-3b77bbf78b6a" containerName="istio-proxy"
Jul 14 12:05:58.043981 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:58.043959    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0135d283-677488f497-xmv59"
Jul 14 12:05:58.046853 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:58.046836    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 14 12:05:58.048010 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:58.047992    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-0135d283-serving-cert\""
Jul 14 12:05:58.048098 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:58.048042    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-62k4x\""
Jul 14 12:05:58.048098 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:58.048052    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 14 12:05:58.055405 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:58.055383    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0135d283-677488f497-xmv59"]
Jul 14 12:05:58.056125 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:58.056107    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:05:58.124437 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:58.124408    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/453ae71d-ac8e-4c81-aba3-960468235ee7-openshift-service-ca\") pod \"maas-api-e2e-ait-0135d283-677488f497-xmv59\" (UID: \"453ae71d-ac8e-4c81-aba3-960468235ee7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0135d283-677488f497-xmv59"
Jul 14 12:05:58.124437 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:58.124439    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxxpb\" (UniqueName: \"kubernetes.io/projected/453ae71d-ac8e-4c81-aba3-960468235ee7-kube-api-access-lxxpb\") pod \"maas-api-e2e-ait-0135d283-677488f497-xmv59\" (UID: \"453ae71d-ac8e-4c81-aba3-960468235ee7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0135d283-677488f497-xmv59"
Jul 14 12:05:58.124649 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:58.124460    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/453ae71d-ac8e-4c81-aba3-960468235ee7-maas-api-tls\") pod \"maas-api-e2e-ait-0135d283-677488f497-xmv59\" (UID: \"453ae71d-ac8e-4c81-aba3-960468235ee7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0135d283-677488f497-xmv59"
Jul 14 12:05:58.153776 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:58.153742    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:05:58.225918 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:58.225832    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/453ae71d-ac8e-4c81-aba3-960468235ee7-openshift-service-ca\") pod \"maas-api-e2e-ait-0135d283-677488f497-xmv59\" (UID: \"453ae71d-ac8e-4c81-aba3-960468235ee7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0135d283-677488f497-xmv59"
Jul 14 12:05:58.225918 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:58.225878    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxxpb\" (UniqueName: \"kubernetes.io/projected/453ae71d-ac8e-4c81-aba3-960468235ee7-kube-api-access-lxxpb\") pod \"maas-api-e2e-ait-0135d283-677488f497-xmv59\" (UID: \"453ae71d-ac8e-4c81-aba3-960468235ee7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0135d283-677488f497-xmv59"
Jul 14 12:05:58.225918 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:58.225910    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/453ae71d-ac8e-4c81-aba3-960468235ee7-maas-api-tls\") pod \"maas-api-e2e-ait-0135d283-677488f497-xmv59\" (UID: \"453ae71d-ac8e-4c81-aba3-960468235ee7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0135d283-677488f497-xmv59"
Jul 14 12:05:58.226169 ip-10-0-136-173 kubenswrapper[2551]: E0714 12:05:58.226026    2551 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-0135d283-serving-cert: secret "maas-api-e2e-ait-0135d283-serving-cert" not found
Jul 14 12:05:58.226169 ip-10-0-136-173 kubenswrapper[2551]: E0714 12:05:58.226096    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/453ae71d-ac8e-4c81-aba3-960468235ee7-maas-api-tls podName:453ae71d-ac8e-4c81-aba3-960468235ee7 nodeName:}" failed. No retries permitted until 2026-07-14 12:05:58.726071634 +0000 UTC m=+1709.957880910 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/453ae71d-ac8e-4c81-aba3-960468235ee7-maas-api-tls") pod "maas-api-e2e-ait-0135d283-677488f497-xmv59" (UID: "453ae71d-ac8e-4c81-aba3-960468235ee7") : secret "maas-api-e2e-ait-0135d283-serving-cert" not found
Jul 14 12:05:58.226524 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:58.226501    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/453ae71d-ac8e-4c81-aba3-960468235ee7-openshift-service-ca\") pod \"maas-api-e2e-ait-0135d283-677488f497-xmv59\" (UID: \"453ae71d-ac8e-4c81-aba3-960468235ee7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0135d283-677488f497-xmv59"
Jul 14 12:05:58.236735 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:58.236712    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxxpb\" (UniqueName: \"kubernetes.io/projected/453ae71d-ac8e-4c81-aba3-960468235ee7-kube-api-access-lxxpb\") pod \"maas-api-e2e-ait-0135d283-677488f497-xmv59\" (UID: \"453ae71d-ac8e-4c81-aba3-960468235ee7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0135d283-677488f497-xmv59"
Jul 14 12:05:58.730595 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:58.730541    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/453ae71d-ac8e-4c81-aba3-960468235ee7-maas-api-tls\") pod \"maas-api-e2e-ait-0135d283-677488f497-xmv59\" (UID: \"453ae71d-ac8e-4c81-aba3-960468235ee7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0135d283-677488f497-xmv59"
Jul 14 12:05:58.730755 ip-10-0-136-173 kubenswrapper[2551]: E0714 12:05:58.730684    2551 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-0135d283-serving-cert: secret "maas-api-e2e-ait-0135d283-serving-cert" not found
Jul 14 12:05:58.730801 ip-10-0-136-173 kubenswrapper[2551]: E0714 12:05:58.730758    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/453ae71d-ac8e-4c81-aba3-960468235ee7-maas-api-tls podName:453ae71d-ac8e-4c81-aba3-960468235ee7 nodeName:}" failed. No retries permitted until 2026-07-14 12:05:59.730742628 +0000 UTC m=+1710.962551903 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/453ae71d-ac8e-4c81-aba3-960468235ee7-maas-api-tls") pod "maas-api-e2e-ait-0135d283-677488f497-xmv59" (UID: "453ae71d-ac8e-4c81-aba3-960468235ee7") : secret "maas-api-e2e-ait-0135d283-serving-cert" not found
Jul 14 12:05:58.812023 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:58.811985    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:05:58.908639 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:58.908610    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:05:58.973081 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:58.973048    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:05:59.738990 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:59.738949    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/453ae71d-ac8e-4c81-aba3-960468235ee7-maas-api-tls\") pod \"maas-api-e2e-ait-0135d283-677488f497-xmv59\" (UID: \"453ae71d-ac8e-4c81-aba3-960468235ee7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0135d283-677488f497-xmv59"
Jul 14 12:05:59.741326 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:59.741301    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/453ae71d-ac8e-4c81-aba3-960468235ee7-maas-api-tls\") pod \"maas-api-e2e-ait-0135d283-677488f497-xmv59\" (UID: \"453ae71d-ac8e-4c81-aba3-960468235ee7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0135d283-677488f497-xmv59"
Jul 14 12:05:59.855877 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:05:59.855837    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0135d283-677488f497-xmv59"
Jul 14 12:06:00.000147 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:06:00.000077    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0135d283-677488f497-xmv59"]
Jul 14 12:06:00.003796 ip-10-0-136-173 kubenswrapper[2551]: W0714 12:06:00.003763    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod453ae71d_ac8e_4c81_aba3_960468235ee7.slice/crio-bf9a8f0aa5faa9238ebe063d62f3d35d242181e7ccc063926b774a33d05d8c62 WatchSource:0}: Error finding container bf9a8f0aa5faa9238ebe063d62f3d35d242181e7ccc063926b774a33d05d8c62: Status 404 returned error can't find the container with id bf9a8f0aa5faa9238ebe063d62f3d35d242181e7ccc063926b774a33d05d8c62
Jul 14 12:06:00.143853 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:06:00.143823    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:06:00.774233 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:06:00.774194    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0135d283-677488f497-xmv59" event={"ID":"453ae71d-ac8e-4c81-aba3-960468235ee7","Type":"ContainerStarted","Data":"b39915b91edc7859be66feef14c60e8df85845b1902ca1c16112bcee5eef6064"}
Jul 14 12:06:00.774398 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:06:00.774276    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0135d283-677488f497-xmv59" event={"ID":"453ae71d-ac8e-4c81-aba3-960468235ee7","Type":"ContainerStarted","Data":"bf9a8f0aa5faa9238ebe063d62f3d35d242181e7ccc063926b774a33d05d8c62"}
Jul 14 12:06:00.774398 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:06:00.774336    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0135d283-677488f497-xmv59"
Jul 14 12:06:00.797260 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:06:00.797169    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0135d283-677488f497-xmv59" podStartSLOduration=2.274351608 podStartE2EDuration="2.797154275s" totalImagesPullingTime="522.802667ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 12:05:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 12:06:00.79575332 +0000 UTC m=+1712.027562616" watchObservedRunningTime="2026-07-14 12:06:00.797154275 +0000 UTC m=+1712.028963572"
Jul 14 12:06:05.490268 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:06:05.490235    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 12:06:06.784388 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:06:06.784361    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0135d283-677488f497-xmv59"
Jul 14 12:06:26.067694 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:06:26.067644    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:06:28.173527 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:06:28.173496    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:06:33.741464 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:06:33.741397    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:06:33.858304 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:06:33.858236    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:07:13.694762 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:07:13.694728    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0135d283-677488f497-xmv59"]
Jul 14 12:07:13.695300 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:07:13.694955    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0135d283-677488f497-xmv59" podUID="453ae71d-ac8e-4c81-aba3-960468235ee7" containerName="maas-api" containerID="cri-o://b39915b91edc7859be66feef14c60e8df85845b1902ca1c16112bcee5eef6064" gracePeriod=30
Jul 14 12:07:13.734969 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:07:13.734934    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:07:13.947356 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:07:13.947300    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0135d283-677488f497-xmv59"
Jul 14 12:07:13.990395 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:07:13.990360    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/453ae71d-ac8e-4c81-aba3-960468235ee7-openshift-service-ca\") pod \"453ae71d-ac8e-4c81-aba3-960468235ee7\" (UID: \"453ae71d-ac8e-4c81-aba3-960468235ee7\") "
Jul 14 12:07:13.990592 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:07:13.990440    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/453ae71d-ac8e-4c81-aba3-960468235ee7-maas-api-tls\") pod \"453ae71d-ac8e-4c81-aba3-960468235ee7\" (UID: \"453ae71d-ac8e-4c81-aba3-960468235ee7\") "
Jul 14 12:07:13.990592 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:07:13.990458    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lxxpb\" (UniqueName: \"kubernetes.io/projected/453ae71d-ac8e-4c81-aba3-960468235ee7-kube-api-access-lxxpb\") pod \"453ae71d-ac8e-4c81-aba3-960468235ee7\" (UID: \"453ae71d-ac8e-4c81-aba3-960468235ee7\") "
Jul 14 12:07:13.990792 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:07:13.990754    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/453ae71d-ac8e-4c81-aba3-960468235ee7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "453ae71d-ac8e-4c81-aba3-960468235ee7" (UID: "453ae71d-ac8e-4c81-aba3-960468235ee7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 12:07:13.992590 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:07:13.992551    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/453ae71d-ac8e-4c81-aba3-960468235ee7-kube-api-access-lxxpb" (OuterVolumeSpecName: "kube-api-access-lxxpb") pod "453ae71d-ac8e-4c81-aba3-960468235ee7" (UID: "453ae71d-ac8e-4c81-aba3-960468235ee7"). InnerVolumeSpecName "kube-api-access-lxxpb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 12:07:13.992753 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:07:13.992735    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/453ae71d-ac8e-4c81-aba3-960468235ee7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "453ae71d-ac8e-4c81-aba3-960468235ee7" (UID: "453ae71d-ac8e-4c81-aba3-960468235ee7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 12:07:14.091480 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:07:14.091452    2551 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/453ae71d-ac8e-4c81-aba3-960468235ee7-openshift-service-ca\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 12:07:14.091480 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:07:14.091477    2551 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/453ae71d-ac8e-4c81-aba3-960468235ee7-maas-api-tls\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 12:07:14.091480 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:07:14.091486    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxxpb\" (UniqueName: \"kubernetes.io/projected/453ae71d-ac8e-4c81-aba3-960468235ee7-kube-api-access-lxxpb\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 12:07:14.119387 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:07:14.119357    2551 generic.go:356] "Generic (PLEG): container finished" podID="453ae71d-ac8e-4c81-aba3-960468235ee7" containerID="b39915b91edc7859be66feef14c60e8df85845b1902ca1c16112bcee5eef6064" exitCode=0
Jul 14 12:07:14.119504 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:07:14.119405    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0135d283-677488f497-xmv59"
Jul 14 12:07:14.119504 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:07:14.119439    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0135d283-677488f497-xmv59" event={"ID":"453ae71d-ac8e-4c81-aba3-960468235ee7","Type":"ContainerDied","Data":"b39915b91edc7859be66feef14c60e8df85845b1902ca1c16112bcee5eef6064"}
Jul 14 12:07:14.119504 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:07:14.119493    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0135d283-677488f497-xmv59" event={"ID":"453ae71d-ac8e-4c81-aba3-960468235ee7","Type":"ContainerDied","Data":"bf9a8f0aa5faa9238ebe063d62f3d35d242181e7ccc063926b774a33d05d8c62"}
Jul 14 12:07:14.119620 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:07:14.119514    2551 scope.go:117] "RemoveContainer" containerID="b39915b91edc7859be66feef14c60e8df85845b1902ca1c16112bcee5eef6064"
Jul 14 12:07:14.129026 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:07:14.129009    2551 scope.go:117] "RemoveContainer" containerID="b39915b91edc7859be66feef14c60e8df85845b1902ca1c16112bcee5eef6064"
Jul 14 12:07:14.129285 ip-10-0-136-173 kubenswrapper[2551]: E0714 12:07:14.129265    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b39915b91edc7859be66feef14c60e8df85845b1902ca1c16112bcee5eef6064\": container with ID starting with b39915b91edc7859be66feef14c60e8df85845b1902ca1c16112bcee5eef6064 not found: ID does not exist" containerID="b39915b91edc7859be66feef14c60e8df85845b1902ca1c16112bcee5eef6064"
Jul 14 12:07:14.129350 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:07:14.129292    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b39915b91edc7859be66feef14c60e8df85845b1902ca1c16112bcee5eef6064"} err="failed to get container status \"b39915b91edc7859be66feef14c60e8df85845b1902ca1c16112bcee5eef6064\": rpc error: code = NotFound desc = could not find container \"b39915b91edc7859be66feef14c60e8df85845b1902ca1c16112bcee5eef6064\": container with ID starting with b39915b91edc7859be66feef14c60e8df85845b1902ca1c16112bcee5eef6064 not found: ID does not exist"
Jul 14 12:07:14.144987 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:07:14.144963    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0135d283-677488f497-xmv59"]
Jul 14 12:07:14.149725 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:07:14.149707    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0135d283-677488f497-xmv59"]
Jul 14 12:07:15.191867 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:07:15.191832    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="453ae71d-ac8e-4c81-aba3-960468235ee7" path="/var/lib/kubelet/pods/453ae71d-ac8e-4c81-aba3-960468235ee7/volumes"
Jul 14 12:07:16.598064 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:07:16.598026    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 12:07:18.177963 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:07:18.177922    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:07:24.884402 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:07:24.884295    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:07:24.960106 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:07:24.960071    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:07:29.192392 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:07:29.192363    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cp57b_62c98a11-ece4-4497-99e7-0840a7626a56/ovn-acl-logging/0.log"
Jul 14 12:07:29.196731 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:07:29.196709    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cp57b_62c98a11-ece4-4497-99e7-0840a7626a56/ovn-acl-logging/0.log"
Jul 14 12:07:29.241242 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:07:29.241208    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:07:29.570148 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:07:29.570114    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:07:30.726831 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:07:30.726797    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:07:30.794411 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:07:30.794367    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-91627dc1-ddf5cc47-cp2bj"]
Jul 14 12:07:30.794816 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:07:30.794796    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="453ae71d-ac8e-4c81-aba3-960468235ee7" containerName="maas-api"
Jul 14 12:07:30.794912 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:07:30.794819    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="453ae71d-ac8e-4c81-aba3-960468235ee7" containerName="maas-api"
Jul 14 12:07:30.794912 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:07:30.794901    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="453ae71d-ac8e-4c81-aba3-960468235ee7" containerName="maas-api"
Jul 14 12:07:30.799037 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:07:30.799016    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-91627dc1-ddf5cc47-cp2bj"
Jul 14 12:07:30.801651 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:07:30.801630    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-62k4x\""
Jul 14 12:07:30.801741 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:07:30.801630    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 14 12:07:30.802659 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:07:30.802639    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 14 12:07:30.802750 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:07:30.802629    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-91627dc1-serving-cert\""
Jul 14 12:07:30.810780 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:07:30.810754    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-91627dc1-ddf5cc47-cp2bj"]
Jul 14 12:07:30.814926 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:07:30.814907    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:07:30.907384 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:07:30.907354    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:07:30.936088 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:07:30.936049    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9530c99e-8903-458b-9af7-0587f5cd7263-maas-api-tls\") pod \"maas-api-e2e-derive-91627dc1-ddf5cc47-cp2bj\" (UID: \"9530c99e-8903-458b-9af7-0587f5cd7263\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-91627dc1-ddf5cc47-cp2bj"
Jul 14 12:07:30.936275 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:07:30.936160    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9530c99e-8903-458b-9af7-0587f5cd7263-openshift-service-ca\") pod \"maas-api-e2e-derive-91627dc1-ddf5cc47-cp2bj\" (UID: \"9530c99e-8903-458b-9af7-0587f5cd7263\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-91627dc1-ddf5cc47-cp2bj"
Jul 14 12:07:30.936275 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:07:30.936195    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfb4v\" (UniqueName: \"kubernetes.io/projected/9530c99e-8903-458b-9af7-0587f5cd7263-kube-api-access-jfb4v\") pod \"maas-api-e2e-derive-91627dc1-ddf5cc47-cp2bj\" (UID: \"9530c99e-8903-458b-9af7-0587f5cd7263\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-91627dc1-ddf5cc47-cp2bj"
Jul 14 12:07:31.000977 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:07:31.000875    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:07:31.037183 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:07:31.037152    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9530c99e-8903-458b-9af7-0587f5cd7263-openshift-service-ca\") pod \"maas-api-e2e-derive-91627dc1-ddf5cc47-cp2bj\" (UID: \"9530c99e-8903-458b-9af7-0587f5cd7263\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-91627dc1-ddf5cc47-cp2bj"
Jul 14 12:07:31.037183 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:07:31.037183    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfb4v\" (UniqueName: \"kubernetes.io/projected/9530c99e-8903-458b-9af7-0587f5cd7263-kube-api-access-jfb4v\") pod \"maas-api-e2e-derive-91627dc1-ddf5cc47-cp2bj\" (UID: \"9530c99e-8903-458b-9af7-0587f5cd7263\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-91627dc1-ddf5cc47-cp2bj"
Jul 14 12:07:31.037376 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:07:31.037228    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9530c99e-8903-458b-9af7-0587f5cd7263-maas-api-tls\") pod \"maas-api-e2e-derive-91627dc1-ddf5cc47-cp2bj\" (UID: \"9530c99e-8903-458b-9af7-0587f5cd7263\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-91627dc1-ddf5cc47-cp2bj"
Jul 14 12:07:31.037848 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:07:31.037828    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9530c99e-8903-458b-9af7-0587f5cd7263-openshift-service-ca\") pod \"maas-api-e2e-derive-91627dc1-ddf5cc47-cp2bj\" (UID: \"9530c99e-8903-458b-9af7-0587f5cd7263\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-91627dc1-ddf5cc47-cp2bj"
Jul 14 12:07:31.039683 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:07:31.039662    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9530c99e-8903-458b-9af7-0587f5cd7263-maas-api-tls\") pod \"maas-api-e2e-derive-91627dc1-ddf5cc47-cp2bj\" (UID: \"9530c99e-8903-458b-9af7-0587f5cd7263\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-91627dc1-ddf5cc47-cp2bj"
Jul 14 12:07:31.048383 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:07:31.048358    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfb4v\" (UniqueName: \"kubernetes.io/projected/9530c99e-8903-458b-9af7-0587f5cd7263-kube-api-access-jfb4v\") pod \"maas-api-e2e-derive-91627dc1-ddf5cc47-cp2bj\" (UID: \"9530c99e-8903-458b-9af7-0587f5cd7263\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-91627dc1-ddf5cc47-cp2bj"
Jul 14 12:07:31.109764 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:07:31.109723    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-91627dc1-ddf5cc47-cp2bj"
Jul 14 12:07:31.270622 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:07:31.270527    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-91627dc1-ddf5cc47-cp2bj"]
Jul 14 12:07:31.274037 ip-10-0-136-173 kubenswrapper[2551]: W0714 12:07:31.274003    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9530c99e_8903_458b_9af7_0587f5cd7263.slice/crio-451a2a5c1124c91aba0d2ab9e6dcd430b0932ceacb57f55c601910a5bb7ac954 WatchSource:0}: Error finding container 451a2a5c1124c91aba0d2ab9e6dcd430b0932ceacb57f55c601910a5bb7ac954: Status 404 returned error can't find the container with id 451a2a5c1124c91aba0d2ab9e6dcd430b0932ceacb57f55c601910a5bb7ac954
Jul 14 12:07:31.523315 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:07:31.523221    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:07:31.622848 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:07:31.622814    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:07:32.211794 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:07:32.211759    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-91627dc1-ddf5cc47-cp2bj" event={"ID":"9530c99e-8903-458b-9af7-0587f5cd7263","Type":"ContainerStarted","Data":"3731a3462e29cb294f734bb956f080605057eed87ddbd3ae51304a22faedad0d"}
Jul 14 12:07:32.212195 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:07:32.211833    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-91627dc1-ddf5cc47-cp2bj" event={"ID":"9530c99e-8903-458b-9af7-0587f5cd7263","Type":"ContainerStarted","Data":"451a2a5c1124c91aba0d2ab9e6dcd430b0932ceacb57f55c601910a5bb7ac954"}
Jul 14 12:07:32.212195 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:07:32.211869    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-91627dc1-ddf5cc47-cp2bj"
Jul 14 12:07:32.237910 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:07:32.237854    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-91627dc1-ddf5cc47-cp2bj" podStartSLOduration=1.841896864 podStartE2EDuration="2.237837905s" totalImagesPullingTime="395.941041ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 12:07:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 12:07:32.23669927 +0000 UTC m=+1803.468508589" watchObservedRunningTime="2026-07-14 12:07:32.237837905 +0000 UTC m=+1803.469647246"
Jul 14 12:07:38.222226 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:07:38.222197    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-91627dc1-ddf5cc47-cp2bj"
Jul 14 12:07:39.097982 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:07:39.097947    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 12:07:46.016207 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:07:46.016176    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:07:46.117458 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:07:46.117418    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:07:49.212920 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:07:49.212880    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:07:54.428075 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:07:54.428031    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:07:54.570675 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:07:54.570625    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:08:35.637617 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:08:35.637566    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-91627dc1-ddf5cc47-cp2bj"]
Jul 14 12:08:35.638058 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:08:35.637772    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-91627dc1-ddf5cc47-cp2bj" podUID="9530c99e-8903-458b-9af7-0587f5cd7263" containerName="maas-api" containerID="cri-o://3731a3462e29cb294f734bb956f080605057eed87ddbd3ae51304a22faedad0d" gracePeriod=30
Jul 14 12:08:35.684339 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:08:35.684314    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:08:35.882778 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:08:35.882757    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-91627dc1-ddf5cc47-cp2bj"
Jul 14 12:08:35.978549 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:08:35.978523    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9530c99e-8903-458b-9af7-0587f5cd7263-openshift-service-ca\") pod \"9530c99e-8903-458b-9af7-0587f5cd7263\" (UID: \"9530c99e-8903-458b-9af7-0587f5cd7263\") "
Jul 14 12:08:35.978705 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:08:35.978561    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9530c99e-8903-458b-9af7-0587f5cd7263-maas-api-tls\") pod \"9530c99e-8903-458b-9af7-0587f5cd7263\" (UID: \"9530c99e-8903-458b-9af7-0587f5cd7263\") "
Jul 14 12:08:35.978705 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:08:35.978637    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jfb4v\" (UniqueName: \"kubernetes.io/projected/9530c99e-8903-458b-9af7-0587f5cd7263-kube-api-access-jfb4v\") pod \"9530c99e-8903-458b-9af7-0587f5cd7263\" (UID: \"9530c99e-8903-458b-9af7-0587f5cd7263\") "
Jul 14 12:08:35.978929 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:08:35.978910    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9530c99e-8903-458b-9af7-0587f5cd7263-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9530c99e-8903-458b-9af7-0587f5cd7263" (UID: "9530c99e-8903-458b-9af7-0587f5cd7263"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 12:08:35.980567 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:08:35.980538    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9530c99e-8903-458b-9af7-0587f5cd7263-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9530c99e-8903-458b-9af7-0587f5cd7263" (UID: "9530c99e-8903-458b-9af7-0587f5cd7263"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 12:08:35.980675 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:08:35.980628    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9530c99e-8903-458b-9af7-0587f5cd7263-kube-api-access-jfb4v" (OuterVolumeSpecName: "kube-api-access-jfb4v") pod "9530c99e-8903-458b-9af7-0587f5cd7263" (UID: "9530c99e-8903-458b-9af7-0587f5cd7263"). InnerVolumeSpecName "kube-api-access-jfb4v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 12:08:36.079408 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:08:36.079378    2551 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9530c99e-8903-458b-9af7-0587f5cd7263-maas-api-tls\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 12:08:36.079408 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:08:36.079404    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfb4v\" (UniqueName: \"kubernetes.io/projected/9530c99e-8903-458b-9af7-0587f5cd7263-kube-api-access-jfb4v\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 12:08:36.079408 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:08:36.079413    2551 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9530c99e-8903-458b-9af7-0587f5cd7263-openshift-service-ca\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 12:08:36.500065 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:08:36.500033    2551 generic.go:356] "Generic (PLEG): container finished" podID="9530c99e-8903-458b-9af7-0587f5cd7263" containerID="3731a3462e29cb294f734bb956f080605057eed87ddbd3ae51304a22faedad0d" exitCode=0
Jul 14 12:08:36.500243 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:08:36.500081    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-91627dc1-ddf5cc47-cp2bj"
Jul 14 12:08:36.500243 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:08:36.500117    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-91627dc1-ddf5cc47-cp2bj" event={"ID":"9530c99e-8903-458b-9af7-0587f5cd7263","Type":"ContainerDied","Data":"3731a3462e29cb294f734bb956f080605057eed87ddbd3ae51304a22faedad0d"}
Jul 14 12:08:36.500243 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:08:36.500172    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-91627dc1-ddf5cc47-cp2bj" event={"ID":"9530c99e-8903-458b-9af7-0587f5cd7263","Type":"ContainerDied","Data":"451a2a5c1124c91aba0d2ab9e6dcd430b0932ceacb57f55c601910a5bb7ac954"}
Jul 14 12:08:36.500243 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:08:36.500200    2551 scope.go:117] "RemoveContainer" containerID="3731a3462e29cb294f734bb956f080605057eed87ddbd3ae51304a22faedad0d"
Jul 14 12:08:36.509296 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:08:36.509277    2551 scope.go:117] "RemoveContainer" containerID="3731a3462e29cb294f734bb956f080605057eed87ddbd3ae51304a22faedad0d"
Jul 14 12:08:36.509523 ip-10-0-136-173 kubenswrapper[2551]: E0714 12:08:36.509505    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3731a3462e29cb294f734bb956f080605057eed87ddbd3ae51304a22faedad0d\": container with ID starting with 3731a3462e29cb294f734bb956f080605057eed87ddbd3ae51304a22faedad0d not found: ID does not exist" containerID="3731a3462e29cb294f734bb956f080605057eed87ddbd3ae51304a22faedad0d"
Jul 14 12:08:36.509569 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:08:36.509532    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3731a3462e29cb294f734bb956f080605057eed87ddbd3ae51304a22faedad0d"} err="failed to get container status \"3731a3462e29cb294f734bb956f080605057eed87ddbd3ae51304a22faedad0d\": rpc error: code = NotFound desc = could not find container \"3731a3462e29cb294f734bb956f080605057eed87ddbd3ae51304a22faedad0d\": container with ID starting with 3731a3462e29cb294f734bb956f080605057eed87ddbd3ae51304a22faedad0d not found: ID does not exist"
Jul 14 12:08:36.524427 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:08:36.524407    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-91627dc1-ddf5cc47-cp2bj"]
Jul 14 12:08:36.530555 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:08:36.530532    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-91627dc1-ddf5cc47-cp2bj"]
Jul 14 12:08:37.191440 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:08:37.191405    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9530c99e-8903-458b-9af7-0587f5cd7263" path="/var/lib/kubelet/pods/9530c99e-8903-458b-9af7-0587f5cd7263/volumes"
Jul 14 12:08:38.295632 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:08:38.295601    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 12:08:40.219571 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:08:40.219538    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:08:46.975275 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:08:46.975241    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:08:47.110763 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:08:47.110728    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:08:50.497321 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:08:50.497248    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:08:51.062905 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:08:51.062870    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:08:51.176079 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:08:51.176051    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:08:51.272840 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:08:51.272800    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:08:52.713678 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:08:52.713646    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:08:56.875190 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:08:56.875152    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:08:59.953717 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:08:59.953686    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:09:00.087537 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:09:00.087507    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:09:00.251810 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:09:00.251719    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:09:00.357735 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:09:00.357703    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:09:00.728904 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:09:00.728865    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:09:20.819625 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:09:20.819566    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:09:26.720516 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:09:26.720475    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:09:34.135524 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:09:34.135490    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:09:34.245460 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:09:34.245423    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:09:34.336312 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:09:34.336280    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:09:50.320230 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:09:50.320191    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:09:57.067039 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:09:57.067000    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:09:57.173964 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:09:57.173928    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:09:57.297975 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:09:57.297942    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:10:07.849292 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:10:07.849252    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:10:08.905520 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:10:08.905483    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:10:15.225177 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:10:15.225142    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:10:15.388975 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:10:15.388938    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:10:15.531942 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:10:15.531862    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:10:16.301416 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:10:16.301367    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:10:16.400075 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:10:16.400042    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:10:16.529211 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:10:16.529158    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:10:17.786684 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:10:17.786648    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 12:10:18.184836 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:10:18.184806    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:10:24.555416 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:10:24.555323    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:10:24.658941 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:10:24.658899    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:10:25.199242 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:10:25.199169    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 12:10:25.265685 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:10:25.265652    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:10:32.340345 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:10:32.340309    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:10:32.476877 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:10:32.476846    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:10:32.617925 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:10:32.617828    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:10:33.019676 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:10:33.019640    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:10:33.119976 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:10:33.119932    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:10:33.303779 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:10:33.303691    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 12:10:35.832463 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:10:35.832426    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:10:35.939199 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:10:35.939164    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5374a9b0-86bccd688c-9rqzq"]
Jul 14 12:10:35.939515 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:10:35.939503    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9530c99e-8903-458b-9af7-0587f5cd7263" containerName="maas-api"
Jul 14 12:10:35.939559 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:10:35.939517    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="9530c99e-8903-458b-9af7-0587f5cd7263" containerName="maas-api"
Jul 14 12:10:35.939615 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:10:35.939597    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="9530c99e-8903-458b-9af7-0587f5cd7263" containerName="maas-api"
Jul 14 12:10:35.942554 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:10:35.942536    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5374a9b0-86bccd688c-9rqzq"
Jul 14 12:10:35.945636 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:10:35.945610    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 14 12:10:35.945788 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:10:35.945750    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-5374a9b0-serving-cert\""
Jul 14 12:10:35.945788 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:10:35.945769    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 14 12:10:35.946749 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:10:35.946730    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-62k4x\""
Jul 14 12:10:35.953063 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:10:35.953042    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5374a9b0-86bccd688c-9rqzq"]
Jul 14 12:10:35.972476 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:10:35.972449    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:10:36.075077 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:10:36.075032    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:10:36.107243 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:10:36.107143    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd8ca4b6-f7ea-488f-8f82-d43500b99e03-maas-api-tls\") pod \"maas-api-e2e-mt-5374a9b0-86bccd688c-9rqzq\" (UID: \"fd8ca4b6-f7ea-488f-8f82-d43500b99e03\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5374a9b0-86bccd688c-9rqzq"
Jul 14 12:10:36.107243 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:10:36.107210    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdpk6\" (UniqueName: \"kubernetes.io/projected/fd8ca4b6-f7ea-488f-8f82-d43500b99e03-kube-api-access-mdpk6\") pod \"maas-api-e2e-mt-5374a9b0-86bccd688c-9rqzq\" (UID: \"fd8ca4b6-f7ea-488f-8f82-d43500b99e03\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5374a9b0-86bccd688c-9rqzq"
Jul 14 12:10:36.107465 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:10:36.107277    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd8ca4b6-f7ea-488f-8f82-d43500b99e03-openshift-service-ca\") pod \"maas-api-e2e-mt-5374a9b0-86bccd688c-9rqzq\" (UID: \"fd8ca4b6-f7ea-488f-8f82-d43500b99e03\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5374a9b0-86bccd688c-9rqzq"
Jul 14 12:10:36.168434 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:10:36.168401    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:10:36.208802 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:10:36.208762    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd8ca4b6-f7ea-488f-8f82-d43500b99e03-maas-api-tls\") pod \"maas-api-e2e-mt-5374a9b0-86bccd688c-9rqzq\" (UID: \"fd8ca4b6-f7ea-488f-8f82-d43500b99e03\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5374a9b0-86bccd688c-9rqzq"
Jul 14 12:10:36.208970 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:10:36.208852    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdpk6\" (UniqueName: \"kubernetes.io/projected/fd8ca4b6-f7ea-488f-8f82-d43500b99e03-kube-api-access-mdpk6\") pod \"maas-api-e2e-mt-5374a9b0-86bccd688c-9rqzq\" (UID: \"fd8ca4b6-f7ea-488f-8f82-d43500b99e03\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5374a9b0-86bccd688c-9rqzq"
Jul 14 12:10:36.208970 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:10:36.208889    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd8ca4b6-f7ea-488f-8f82-d43500b99e03-openshift-service-ca\") pod \"maas-api-e2e-mt-5374a9b0-86bccd688c-9rqzq\" (UID: \"fd8ca4b6-f7ea-488f-8f82-d43500b99e03\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5374a9b0-86bccd688c-9rqzq"
Jul 14 12:10:36.209685 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:10:36.209658    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd8ca4b6-f7ea-488f-8f82-d43500b99e03-openshift-service-ca\") pod \"maas-api-e2e-mt-5374a9b0-86bccd688c-9rqzq\" (UID: \"fd8ca4b6-f7ea-488f-8f82-d43500b99e03\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5374a9b0-86bccd688c-9rqzq"
Jul 14 12:10:36.211431 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:10:36.211411    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd8ca4b6-f7ea-488f-8f82-d43500b99e03-maas-api-tls\") pod \"maas-api-e2e-mt-5374a9b0-86bccd688c-9rqzq\" (UID: \"fd8ca4b6-f7ea-488f-8f82-d43500b99e03\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5374a9b0-86bccd688c-9rqzq"
Jul 14 12:10:36.219428 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:10:36.219406    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdpk6\" (UniqueName: \"kubernetes.io/projected/fd8ca4b6-f7ea-488f-8f82-d43500b99e03-kube-api-access-mdpk6\") pod \"maas-api-e2e-mt-5374a9b0-86bccd688c-9rqzq\" (UID: \"fd8ca4b6-f7ea-488f-8f82-d43500b99e03\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5374a9b0-86bccd688c-9rqzq"
Jul 14 12:10:36.253249 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:10:36.253215    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5374a9b0-86bccd688c-9rqzq"
Jul 14 12:10:36.400728 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:10:36.400700    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5374a9b0-86bccd688c-9rqzq"]
Jul 14 12:10:36.402924 ip-10-0-136-173 kubenswrapper[2551]: W0714 12:10:36.402882    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd8ca4b6_f7ea_488f_8f82_d43500b99e03.slice/crio-b723d3a6e832c0a0ef08beda695d5f848745482fafd5e4d092d3c49167ce8fec WatchSource:0}: Error finding container b723d3a6e832c0a0ef08beda695d5f848745482fafd5e4d092d3c49167ce8fec: Status 404 returned error can't find the container with id b723d3a6e832c0a0ef08beda695d5f848745482fafd5e4d092d3c49167ce8fec
Jul 14 12:10:36.404268 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:10:36.404250    2551 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 14 12:10:36.582247 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:10:36.582195    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:10:36.691760 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:10:36.691726    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:10:37.045267 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:10:37.045234    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5374a9b0-86bccd688c-9rqzq" event={"ID":"fd8ca4b6-f7ea-488f-8f82-d43500b99e03","Type":"ContainerStarted","Data":"53de479c1ef002761c08488721713b3462f17abc0b1730bba7fc0c347f15e41c"}
Jul 14 12:10:37.045776 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:10:37.045292    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5374a9b0-86bccd688c-9rqzq" event={"ID":"fd8ca4b6-f7ea-488f-8f82-d43500b99e03","Type":"ContainerStarted","Data":"b723d3a6e832c0a0ef08beda695d5f848745482fafd5e4d092d3c49167ce8fec"}
Jul 14 12:10:37.045776 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:10:37.045332    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5374a9b0-86bccd688c-9rqzq"
Jul 14 12:10:37.069386 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:10:37.069325    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5374a9b0-86bccd688c-9rqzq" podStartSLOduration=1.571915927 podStartE2EDuration="2.069310263s" totalImagesPullingTime="497.394336ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 12:10:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 12:10:37.067880735 +0000 UTC m=+1988.299690032" watchObservedRunningTime="2026-07-14 12:10:37.069310263 +0000 UTC m=+1988.301119559"
Jul 14 12:10:40.360610 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:10:40.360548    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:10:41.076183 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:10:41.076149    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:10:41.182258 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:10:41.182226    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:10:41.535103 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:10:41.535059    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:10:43.055742 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:10:43.055705    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5374a9b0-86bccd688c-9rqzq"
Jul 14 12:10:45.191899 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:10:45.191869    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 12:10:48.539626 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:10:48.539563    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:10:48.776663 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:10:48.776625    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:10:59.354514 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:10:59.354474    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5374a9b0-86bccd688c-9rqzq"]
Jul 14 12:10:59.355025 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:10:59.354813    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5374a9b0-86bccd688c-9rqzq" podUID="fd8ca4b6-f7ea-488f-8f82-d43500b99e03" containerName="maas-api" containerID="cri-o://53de479c1ef002761c08488721713b3462f17abc0b1730bba7fc0c347f15e41c" gracePeriod=30
Jul 14 12:10:59.386924 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:10:59.386883    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:10:59.620530 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:10:59.620460    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5374a9b0-86bccd688c-9rqzq"
Jul 14 12:10:59.718719 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:10:59.718686    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd8ca4b6-f7ea-488f-8f82-d43500b99e03-maas-api-tls\") pod \"fd8ca4b6-f7ea-488f-8f82-d43500b99e03\" (UID: \"fd8ca4b6-f7ea-488f-8f82-d43500b99e03\") "
Jul 14 12:10:59.718905 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:10:59.718739    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mdpk6\" (UniqueName: \"kubernetes.io/projected/fd8ca4b6-f7ea-488f-8f82-d43500b99e03-kube-api-access-mdpk6\") pod \"fd8ca4b6-f7ea-488f-8f82-d43500b99e03\" (UID: \"fd8ca4b6-f7ea-488f-8f82-d43500b99e03\") "
Jul 14 12:10:59.718905 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:10:59.718775    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd8ca4b6-f7ea-488f-8f82-d43500b99e03-openshift-service-ca\") pod \"fd8ca4b6-f7ea-488f-8f82-d43500b99e03\" (UID: \"fd8ca4b6-f7ea-488f-8f82-d43500b99e03\") "
Jul 14 12:10:59.719201 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:10:59.719167    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd8ca4b6-f7ea-488f-8f82-d43500b99e03-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fd8ca4b6-f7ea-488f-8f82-d43500b99e03" (UID: "fd8ca4b6-f7ea-488f-8f82-d43500b99e03"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 12:10:59.721021 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:10:59.720979    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd8ca4b6-f7ea-488f-8f82-d43500b99e03-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fd8ca4b6-f7ea-488f-8f82-d43500b99e03" (UID: "fd8ca4b6-f7ea-488f-8f82-d43500b99e03"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 12:10:59.721021 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:10:59.720995    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd8ca4b6-f7ea-488f-8f82-d43500b99e03-kube-api-access-mdpk6" (OuterVolumeSpecName: "kube-api-access-mdpk6") pod "fd8ca4b6-f7ea-488f-8f82-d43500b99e03" (UID: "fd8ca4b6-f7ea-488f-8f82-d43500b99e03"). InnerVolumeSpecName "kube-api-access-mdpk6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 12:10:59.820223 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:10:59.820182    2551 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd8ca4b6-f7ea-488f-8f82-d43500b99e03-maas-api-tls\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 12:10:59.820223 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:10:59.820219    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdpk6\" (UniqueName: \"kubernetes.io/projected/fd8ca4b6-f7ea-488f-8f82-d43500b99e03-kube-api-access-mdpk6\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 12:10:59.820223 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:10:59.820231    2551 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd8ca4b6-f7ea-488f-8f82-d43500b99e03-openshift-service-ca\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 12:11:00.153703 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:11:00.153670    2551 generic.go:356] "Generic (PLEG): container finished" podID="fd8ca4b6-f7ea-488f-8f82-d43500b99e03" containerID="53de479c1ef002761c08488721713b3462f17abc0b1730bba7fc0c347f15e41c" exitCode=0
Jul 14 12:11:00.153898 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:11:00.153714    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5374a9b0-86bccd688c-9rqzq"
Jul 14 12:11:00.153898 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:11:00.153723    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5374a9b0-86bccd688c-9rqzq" event={"ID":"fd8ca4b6-f7ea-488f-8f82-d43500b99e03","Type":"ContainerDied","Data":"53de479c1ef002761c08488721713b3462f17abc0b1730bba7fc0c347f15e41c"}
Jul 14 12:11:00.153898 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:11:00.153800    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5374a9b0-86bccd688c-9rqzq" event={"ID":"fd8ca4b6-f7ea-488f-8f82-d43500b99e03","Type":"ContainerDied","Data":"b723d3a6e832c0a0ef08beda695d5f848745482fafd5e4d092d3c49167ce8fec"}
Jul 14 12:11:00.153898 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:11:00.153829    2551 scope.go:117] "RemoveContainer" containerID="53de479c1ef002761c08488721713b3462f17abc0b1730bba7fc0c347f15e41c"
Jul 14 12:11:00.162812 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:11:00.162791    2551 scope.go:117] "RemoveContainer" containerID="53de479c1ef002761c08488721713b3462f17abc0b1730bba7fc0c347f15e41c"
Jul 14 12:11:00.163087 ip-10-0-136-173 kubenswrapper[2551]: E0714 12:11:00.163067    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53de479c1ef002761c08488721713b3462f17abc0b1730bba7fc0c347f15e41c\": container with ID starting with 53de479c1ef002761c08488721713b3462f17abc0b1730bba7fc0c347f15e41c not found: ID does not exist" containerID="53de479c1ef002761c08488721713b3462f17abc0b1730bba7fc0c347f15e41c"
Jul 14 12:11:00.163173 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:11:00.163100    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53de479c1ef002761c08488721713b3462f17abc0b1730bba7fc0c347f15e41c"} err="failed to get container status \"53de479c1ef002761c08488721713b3462f17abc0b1730bba7fc0c347f15e41c\": rpc error: code = NotFound desc = could not find container \"53de479c1ef002761c08488721713b3462f17abc0b1730bba7fc0c347f15e41c\": container with ID starting with 53de479c1ef002761c08488721713b3462f17abc0b1730bba7fc0c347f15e41c not found: ID does not exist"
Jul 14 12:11:00.180950 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:11:00.180908    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5374a9b0-86bccd688c-9rqzq"]
Jul 14 12:11:00.189697 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:11:00.189639    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5374a9b0-86bccd688c-9rqzq"]
Jul 14 12:11:01.191670 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:11:01.191625    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd8ca4b6-f7ea-488f-8f82-d43500b99e03" path="/var/lib/kubelet/pods/fd8ca4b6-f7ea-488f-8f82-d43500b99e03/volumes"
Jul 14 12:11:02.490331 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:11:02.490284    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 12:11:02.605533 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:11:02.605498    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:11:02.684187 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:11:02.684152    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:11:04.013338 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:11:04.013286    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:11:10.591386 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:11:10.591348    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:11:10.669625 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:11:10.669566    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:11:14.741914 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:11:14.741875    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:11:14.974403 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:11:14.974366    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:11:15.087997 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:11:15.087897    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:11:15.188239 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:11:15.188181    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:11:15.340740 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:11:15.340649    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:11:22.284396 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:11:22.284356    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:11:22.388203 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:11:22.388168    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:11:22.496997 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:11:22.496958    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:11:22.659817 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:11:22.659719    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:11:27.047475 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:11:27.047442    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:11:27.143171 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:11:27.143132    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:11:29.315750 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:11:29.315719    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:11:30.804003 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:11:30.803964    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:11:36.362884 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:11:36.362846    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:11:36.611335 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:11:36.611297    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:11:36.917305 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:11:36.917271    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:11:40.043140 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:11:40.043105    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:11:40.077764 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:11:40.077722    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c52b78c6-596b4b7c66-qfr5r"]
Jul 14 12:11:40.078067 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:11:40.078055    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd8ca4b6-f7ea-488f-8f82-d43500b99e03" containerName="maas-api"
Jul 14 12:11:40.078114 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:11:40.078068    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd8ca4b6-f7ea-488f-8f82-d43500b99e03" containerName="maas-api"
Jul 14 12:11:40.078154 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:11:40.078130    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd8ca4b6-f7ea-488f-8f82-d43500b99e03" containerName="maas-api"
Jul 14 12:11:40.080999 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:11:40.080979    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c52b78c6-596b4b7c66-qfr5r"
Jul 14 12:11:40.083712 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:11:40.083688    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-c52b78c6-serving-cert\""
Jul 14 12:11:40.083850 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:11:40.083770    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 14 12:11:40.084854 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:11:40.084836    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-62k4x\""
Jul 14 12:11:40.084898 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:11:40.084876    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 14 12:11:40.100215 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:11:40.100187    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c52b78c6-596b4b7c66-qfr5r"]
Jul 14 12:11:40.170738 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:11:40.170697    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b6156d3-9204-4569-b61d-71fd86b67eba-maas-api-tls\") pod \"maas-api-e2e-mt-c52b78c6-596b4b7c66-qfr5r\" (UID: \"2b6156d3-9204-4569-b61d-71fd86b67eba\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c52b78c6-596b4b7c66-qfr5r"
Jul 14 12:11:40.170940 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:11:40.170788    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b6156d3-9204-4569-b61d-71fd86b67eba-openshift-service-ca\") pod \"maas-api-e2e-mt-c52b78c6-596b4b7c66-qfr5r\" (UID: \"2b6156d3-9204-4569-b61d-71fd86b67eba\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c52b78c6-596b4b7c66-qfr5r"
Jul 14 12:11:40.170940 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:11:40.170816    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhsxd\" (UniqueName: \"kubernetes.io/projected/2b6156d3-9204-4569-b61d-71fd86b67eba-kube-api-access-mhsxd\") pod \"maas-api-e2e-mt-c52b78c6-596b4b7c66-qfr5r\" (UID: \"2b6156d3-9204-4569-b61d-71fd86b67eba\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c52b78c6-596b4b7c66-qfr5r"
Jul 14 12:11:40.272104 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:11:40.272061    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b6156d3-9204-4569-b61d-71fd86b67eba-maas-api-tls\") pod \"maas-api-e2e-mt-c52b78c6-596b4b7c66-qfr5r\" (UID: \"2b6156d3-9204-4569-b61d-71fd86b67eba\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c52b78c6-596b4b7c66-qfr5r"
Jul 14 12:11:40.272293 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:11:40.272156    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b6156d3-9204-4569-b61d-71fd86b67eba-openshift-service-ca\") pod \"maas-api-e2e-mt-c52b78c6-596b4b7c66-qfr5r\" (UID: \"2b6156d3-9204-4569-b61d-71fd86b67eba\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c52b78c6-596b4b7c66-qfr5r"
Jul 14 12:11:40.272293 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:11:40.272187    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhsxd\" (UniqueName: \"kubernetes.io/projected/2b6156d3-9204-4569-b61d-71fd86b67eba-kube-api-access-mhsxd\") pod \"maas-api-e2e-mt-c52b78c6-596b4b7c66-qfr5r\" (UID: \"2b6156d3-9204-4569-b61d-71fd86b67eba\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c52b78c6-596b4b7c66-qfr5r"
Jul 14 12:11:40.272293 ip-10-0-136-173 kubenswrapper[2551]: E0714 12:11:40.272281    2551 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-c52b78c6-serving-cert: secret "maas-api-e2e-mt-c52b78c6-serving-cert" not found
Jul 14 12:11:40.272423 ip-10-0-136-173 kubenswrapper[2551]: E0714 12:11:40.272363    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b6156d3-9204-4569-b61d-71fd86b67eba-maas-api-tls podName:2b6156d3-9204-4569-b61d-71fd86b67eba nodeName:}" failed. No retries permitted until 2026-07-14 12:11:40.772341159 +0000 UTC m=+2052.004150440 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2b6156d3-9204-4569-b61d-71fd86b67eba-maas-api-tls") pod "maas-api-e2e-mt-c52b78c6-596b4b7c66-qfr5r" (UID: "2b6156d3-9204-4569-b61d-71fd86b67eba") : secret "maas-api-e2e-mt-c52b78c6-serving-cert" not found
Jul 14 12:11:40.272891 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:11:40.272871    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b6156d3-9204-4569-b61d-71fd86b67eba-openshift-service-ca\") pod \"maas-api-e2e-mt-c52b78c6-596b4b7c66-qfr5r\" (UID: \"2b6156d3-9204-4569-b61d-71fd86b67eba\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c52b78c6-596b4b7c66-qfr5r"
Jul 14 12:11:40.286613 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:11:40.286572    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:11:40.289843 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:11:40.289817    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhsxd\" (UniqueName: \"kubernetes.io/projected/2b6156d3-9204-4569-b61d-71fd86b67eba-kube-api-access-mhsxd\") pod \"maas-api-e2e-mt-c52b78c6-596b4b7c66-qfr5r\" (UID: \"2b6156d3-9204-4569-b61d-71fd86b67eba\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c52b78c6-596b4b7c66-qfr5r"
Jul 14 12:11:40.506381 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:11:40.506349    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:11:40.747193 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:11:40.747155    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:11:40.777513 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:11:40.777409    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b6156d3-9204-4569-b61d-71fd86b67eba-maas-api-tls\") pod \"maas-api-e2e-mt-c52b78c6-596b4b7c66-qfr5r\" (UID: \"2b6156d3-9204-4569-b61d-71fd86b67eba\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c52b78c6-596b4b7c66-qfr5r"
Jul 14 12:11:40.780017 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:11:40.779986    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b6156d3-9204-4569-b61d-71fd86b67eba-maas-api-tls\") pod \"maas-api-e2e-mt-c52b78c6-596b4b7c66-qfr5r\" (UID: \"2b6156d3-9204-4569-b61d-71fd86b67eba\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c52b78c6-596b4b7c66-qfr5r"
Jul 14 12:11:40.854456 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:11:40.854419    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:11:40.992021 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:11:40.991984    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c52b78c6-596b4b7c66-qfr5r"
Jul 14 12:11:41.145498 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:11:41.145464    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c52b78c6-596b4b7c66-qfr5r"]
Jul 14 12:11:41.148919 ip-10-0-136-173 kubenswrapper[2551]: W0714 12:11:41.148890    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b6156d3_9204_4569_b61d_71fd86b67eba.slice/crio-dce2d371e0565c65d978ed4291df857ea1d8e94658206d5379338276b00d7c6a WatchSource:0}: Error finding container dce2d371e0565c65d978ed4291df857ea1d8e94658206d5379338276b00d7c6a: Status 404 returned error can't find the container with id dce2d371e0565c65d978ed4291df857ea1d8e94658206d5379338276b00d7c6a
Jul 14 12:11:41.359310 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:11:41.359206    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c52b78c6-596b4b7c66-qfr5r" event={"ID":"2b6156d3-9204-4569-b61d-71fd86b67eba","Type":"ContainerStarted","Data":"dce2d371e0565c65d978ed4291df857ea1d8e94658206d5379338276b00d7c6a"}
Jul 14 12:11:42.365397 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:11:42.365363    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c52b78c6-596b4b7c66-qfr5r" event={"ID":"2b6156d3-9204-4569-b61d-71fd86b67eba","Type":"ContainerStarted","Data":"942f7b1893821c1130168d5dc9fb7b653994a206b9bfe23734826e85ebfd2f12"}
Jul 14 12:11:42.365809 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:11:42.365476    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c52b78c6-596b4b7c66-qfr5r"
Jul 14 12:11:42.385784 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:11:42.385730    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c52b78c6-596b4b7c66-qfr5r" podStartSLOduration=1.963951701 podStartE2EDuration="2.385712307s" totalImagesPullingTime="421.760606ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 12:11:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 12:11:42.384231643 +0000 UTC m=+2053.616040938" watchObservedRunningTime="2026-07-14 12:11:42.385712307 +0000 UTC m=+2053.617521602"
Jul 14 12:11:44.487386 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:11:44.487347    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:11:45.710729 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:11:45.710695    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 12:11:48.377386 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:11:48.377351    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c52b78c6-596b4b7c66-qfr5r"
Jul 14 12:11:57.089485 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:11:57.089387    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-c52b78c6/e2e-lifecycle-model-c52b78c6-kserve-5d449b6767-zthkx"]
Jul 14 12:11:57.101217 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:11:57.101190    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-c52b78c6/e2e-lifecycle-model-c52b78c6-kserve-5d449b6767-zthkx"
Jul 14 12:11:57.103736 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:11:57.103705    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-c52b78c6/e2e-lifecycle-model-c52b78c6-kserve-5d449b6767-zthkx"]
Jul 14 12:11:57.105047 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:11:57.105018    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-c52b78c6\"/\"kube-root-ca.crt\""
Jul 14 12:11:57.105200 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:11:57.105070    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-c52b78c6\"/\"default-dockercfg-mr5d5\""
Jul 14 12:11:57.105200 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:11:57.105145    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-c52b78c6\"/\"e2e-lifecycle-model-c52b78c6-kserve-self-signed-certs\""
Jul 14 12:11:57.105200 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:11:57.105028    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-c52b78c6\"/\"openshift-service-ca.crt\""
Jul 14 12:11:57.190988 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:11:57.190950    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:11:57.224167 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:11:57.224129    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/404635b1-e9f8-44a4-81b9-3861f8e69e2a-dshm\") pod \"e2e-lifecycle-model-c52b78c6-kserve-5d449b6767-zthkx\" (UID: \"404635b1-e9f8-44a4-81b9-3861f8e69e2a\") " pod="ai-tenant-e2e-mt-c52b78c6/e2e-lifecycle-model-c52b78c6-kserve-5d449b6767-zthkx"
Jul 14 12:11:57.224167 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:11:57.224187    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/404635b1-e9f8-44a4-81b9-3861f8e69e2a-tls-certs\") pod \"e2e-lifecycle-model-c52b78c6-kserve-5d449b6767-zthkx\" (UID: \"404635b1-e9f8-44a4-81b9-3861f8e69e2a\") " pod="ai-tenant-e2e-mt-c52b78c6/e2e-lifecycle-model-c52b78c6-kserve-5d449b6767-zthkx"
Jul 14 12:11:57.224442 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:11:57.224246    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/404635b1-e9f8-44a4-81b9-3861f8e69e2a-model-cache\") pod \"e2e-lifecycle-model-c52b78c6-kserve-5d449b6767-zthkx\" (UID: \"404635b1-e9f8-44a4-81b9-3861f8e69e2a\") " pod="ai-tenant-e2e-mt-c52b78c6/e2e-lifecycle-model-c52b78c6-kserve-5d449b6767-zthkx"
Jul 14 12:11:57.224442 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:11:57.224288    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/404635b1-e9f8-44a4-81b9-3861f8e69e2a-tmp-dir\") pod \"e2e-lifecycle-model-c52b78c6-kserve-5d449b6767-zthkx\" (UID: \"404635b1-e9f8-44a4-81b9-3861f8e69e2a\") " pod="ai-tenant-e2e-mt-c52b78c6/e2e-lifecycle-model-c52b78c6-kserve-5d449b6767-zthkx"
Jul 14 12:11:57.224442 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:11:57.224381    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/404635b1-e9f8-44a4-81b9-3861f8e69e2a-home\") pod \"e2e-lifecycle-model-c52b78c6-kserve-5d449b6767-zthkx\" (UID: \"404635b1-e9f8-44a4-81b9-3861f8e69e2a\") " pod="ai-tenant-e2e-mt-c52b78c6/e2e-lifecycle-model-c52b78c6-kserve-5d449b6767-zthkx"
Jul 14 12:11:57.224442 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:11:57.224429    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnhrt\" (UniqueName: \"kubernetes.io/projected/404635b1-e9f8-44a4-81b9-3861f8e69e2a-kube-api-access-vnhrt\") pod \"e2e-lifecycle-model-c52b78c6-kserve-5d449b6767-zthkx\" (UID: \"404635b1-e9f8-44a4-81b9-3861f8e69e2a\") " pod="ai-tenant-e2e-mt-c52b78c6/e2e-lifecycle-model-c52b78c6-kserve-5d449b6767-zthkx"
Jul 14 12:11:57.325128 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:11:57.325076    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/404635b1-e9f8-44a4-81b9-3861f8e69e2a-dshm\") pod \"e2e-lifecycle-model-c52b78c6-kserve-5d449b6767-zthkx\" (UID: \"404635b1-e9f8-44a4-81b9-3861f8e69e2a\") " pod="ai-tenant-e2e-mt-c52b78c6/e2e-lifecycle-model-c52b78c6-kserve-5d449b6767-zthkx"
Jul 14 12:11:57.325128 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:11:57.325135    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/404635b1-e9f8-44a4-81b9-3861f8e69e2a-tls-certs\") pod \"e2e-lifecycle-model-c52b78c6-kserve-5d449b6767-zthkx\" (UID: \"404635b1-e9f8-44a4-81b9-3861f8e69e2a\") " pod="ai-tenant-e2e-mt-c52b78c6/e2e-lifecycle-model-c52b78c6-kserve-5d449b6767-zthkx"
Jul 14 12:11:57.325407 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:11:57.325178    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/404635b1-e9f8-44a4-81b9-3861f8e69e2a-model-cache\") pod \"e2e-lifecycle-model-c52b78c6-kserve-5d449b6767-zthkx\" (UID: \"404635b1-e9f8-44a4-81b9-3861f8e69e2a\") " pod="ai-tenant-e2e-mt-c52b78c6/e2e-lifecycle-model-c52b78c6-kserve-5d449b6767-zthkx"
Jul 14 12:11:57.325407 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:11:57.325222    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/404635b1-e9f8-44a4-81b9-3861f8e69e2a-tmp-dir\") pod \"e2e-lifecycle-model-c52b78c6-kserve-5d449b6767-zthkx\" (UID: \"404635b1-e9f8-44a4-81b9-3861f8e69e2a\") " pod="ai-tenant-e2e-mt-c52b78c6/e2e-lifecycle-model-c52b78c6-kserve-5d449b6767-zthkx"
Jul 14 12:11:57.325407 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:11:57.325269    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/404635b1-e9f8-44a4-81b9-3861f8e69e2a-home\") pod \"e2e-lifecycle-model-c52b78c6-kserve-5d449b6767-zthkx\" (UID: \"404635b1-e9f8-44a4-81b9-3861f8e69e2a\") " pod="ai-tenant-e2e-mt-c52b78c6/e2e-lifecycle-model-c52b78c6-kserve-5d449b6767-zthkx"
Jul 14 12:11:57.325407 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:11:57.325323    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnhrt\" (UniqueName: \"kubernetes.io/projected/404635b1-e9f8-44a4-81b9-3861f8e69e2a-kube-api-access-vnhrt\") pod \"e2e-lifecycle-model-c52b78c6-kserve-5d449b6767-zthkx\" (UID: \"404635b1-e9f8-44a4-81b9-3861f8e69e2a\") " pod="ai-tenant-e2e-mt-c52b78c6/e2e-lifecycle-model-c52b78c6-kserve-5d449b6767-zthkx"
Jul 14 12:11:57.325679 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:11:57.325654    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/404635b1-e9f8-44a4-81b9-3861f8e69e2a-model-cache\") pod \"e2e-lifecycle-model-c52b78c6-kserve-5d449b6767-zthkx\" (UID: \"404635b1-e9f8-44a4-81b9-3861f8e69e2a\") " pod="ai-tenant-e2e-mt-c52b78c6/e2e-lifecycle-model-c52b78c6-kserve-5d449b6767-zthkx"
Jul 14 12:11:57.325925 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:11:57.325897    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/404635b1-e9f8-44a4-81b9-3861f8e69e2a-home\") pod \"e2e-lifecycle-model-c52b78c6-kserve-5d449b6767-zthkx\" (UID: \"404635b1-e9f8-44a4-81b9-3861f8e69e2a\") " pod="ai-tenant-e2e-mt-c52b78c6/e2e-lifecycle-model-c52b78c6-kserve-5d449b6767-zthkx"
Jul 14 12:11:57.326104 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:11:57.325945    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/404635b1-e9f8-44a4-81b9-3861f8e69e2a-tmp-dir\") pod \"e2e-lifecycle-model-c52b78c6-kserve-5d449b6767-zthkx\" (UID: \"404635b1-e9f8-44a4-81b9-3861f8e69e2a\") " pod="ai-tenant-e2e-mt-c52b78c6/e2e-lifecycle-model-c52b78c6-kserve-5d449b6767-zthkx"
Jul 14 12:11:57.327933 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:11:57.327912    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/404635b1-e9f8-44a4-81b9-3861f8e69e2a-dshm\") pod \"e2e-lifecycle-model-c52b78c6-kserve-5d449b6767-zthkx\" (UID: \"404635b1-e9f8-44a4-81b9-3861f8e69e2a\") " pod="ai-tenant-e2e-mt-c52b78c6/e2e-lifecycle-model-c52b78c6-kserve-5d449b6767-zthkx"
Jul 14 12:11:57.328117 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:11:57.328093    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/404635b1-e9f8-44a4-81b9-3861f8e69e2a-tls-certs\") pod \"e2e-lifecycle-model-c52b78c6-kserve-5d449b6767-zthkx\" (UID: \"404635b1-e9f8-44a4-81b9-3861f8e69e2a\") " pod="ai-tenant-e2e-mt-c52b78c6/e2e-lifecycle-model-c52b78c6-kserve-5d449b6767-zthkx"
Jul 14 12:11:57.337218 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:11:57.337188    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnhrt\" (UniqueName: \"kubernetes.io/projected/404635b1-e9f8-44a4-81b9-3861f8e69e2a-kube-api-access-vnhrt\") pod \"e2e-lifecycle-model-c52b78c6-kserve-5d449b6767-zthkx\" (UID: \"404635b1-e9f8-44a4-81b9-3861f8e69e2a\") " pod="ai-tenant-e2e-mt-c52b78c6/e2e-lifecycle-model-c52b78c6-kserve-5d449b6767-zthkx"
Jul 14 12:11:57.412300 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:11:57.412206    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-c52b78c6/e2e-lifecycle-model-c52b78c6-kserve-5d449b6767-zthkx"
Jul 14 12:11:57.558446 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:11:57.558408    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-c52b78c6/e2e-lifecycle-model-c52b78c6-kserve-5d449b6767-zthkx"]
Jul 14 12:11:57.561553 ip-10-0-136-173 kubenswrapper[2551]: W0714 12:11:57.561526    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod404635b1_e9f8_44a4_81b9_3861f8e69e2a.slice/crio-52891c02f81519f83eded0e06a33c021532889b2e089000356fcbd1452a0e318 WatchSource:0}: Error finding container 52891c02f81519f83eded0e06a33c021532889b2e089000356fcbd1452a0e318: Status 404 returned error can't find the container with id 52891c02f81519f83eded0e06a33c021532889b2e089000356fcbd1452a0e318
Jul 14 12:11:58.446785 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:11:58.446747    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-c52b78c6/e2e-lifecycle-model-c52b78c6-kserve-5d449b6767-zthkx" event={"ID":"404635b1-e9f8-44a4-81b9-3861f8e69e2a","Type":"ContainerStarted","Data":"8b12dc1dc7f186322c8466585372690709979cb9915ee5d6d7c1f17a6aa12028"}
Jul 14 12:11:58.446785 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:11:58.446792    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-c52b78c6/e2e-lifecycle-model-c52b78c6-kserve-5d449b6767-zthkx" event={"ID":"404635b1-e9f8-44a4-81b9-3861f8e69e2a","Type":"ContainerStarted","Data":"52891c02f81519f83eded0e06a33c021532889b2e089000356fcbd1452a0e318"}
Jul 14 12:11:58.472274 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:11:58.472219    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-c52b78c6/e2e-lifecycle-model-c52b78c6-kserve-5d449b6767-zthkx" podStartSLOduration=1.47219896 podStartE2EDuration="1.47219896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 12:11:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 12:11:58.4704604 +0000 UTC m=+2069.702269697" watchObservedRunningTime="2026-07-14 12:11:58.47219896 +0000 UTC m=+2069.704008255"
Jul 14 12:12:00.286152 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:12:00.286110    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 12:12:05.404865 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:12:05.404824    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:12:07.412992 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:12:07.412944    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-c52b78c6/e2e-lifecycle-model-c52b78c6-kserve-5d449b6767-zthkx"
Jul 14 12:12:07.413511 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:12:07.413017    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-c52b78c6/e2e-lifecycle-model-c52b78c6-kserve-5d449b6767-zthkx"
Jul 14 12:12:07.427233 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:12:07.427184    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-c52b78c6/e2e-lifecycle-model-c52b78c6-kserve-5d449b6767-zthkx"
Jul 14 12:12:07.499728 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:12:07.499697    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-c52b78c6/e2e-lifecycle-model-c52b78c6-kserve-5d449b6767-zthkx"
Jul 14 12:12:15.194504 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:12:15.194468    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 12:12:28.404490 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:12:28.404454    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c52b78c6-596b4b7c66-qfr5r"]
Jul 14 12:12:28.404912 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:12:28.404690    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c52b78c6-596b4b7c66-qfr5r" podUID="2b6156d3-9204-4569-b61d-71fd86b67eba" containerName="maas-api" containerID="cri-o://942f7b1893821c1130168d5dc9fb7b653994a206b9bfe23734826e85ebfd2f12" gracePeriod=30
Jul 14 12:12:28.448451 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:12:28.448397    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:12:28.590653 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:12:28.590616    2551 generic.go:356] "Generic (PLEG): container finished" podID="2b6156d3-9204-4569-b61d-71fd86b67eba" containerID="942f7b1893821c1130168d5dc9fb7b653994a206b9bfe23734826e85ebfd2f12" exitCode=0
Jul 14 12:12:28.590820 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:12:28.590683    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c52b78c6-596b4b7c66-qfr5r" event={"ID":"2b6156d3-9204-4569-b61d-71fd86b67eba","Type":"ContainerDied","Data":"942f7b1893821c1130168d5dc9fb7b653994a206b9bfe23734826e85ebfd2f12"}
Jul 14 12:12:28.663084 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:12:28.662999    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c52b78c6-596b4b7c66-qfr5r"
Jul 14 12:12:28.709745 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:12:28.709704    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mhsxd\" (UniqueName: \"kubernetes.io/projected/2b6156d3-9204-4569-b61d-71fd86b67eba-kube-api-access-mhsxd\") pod \"2b6156d3-9204-4569-b61d-71fd86b67eba\" (UID: \"2b6156d3-9204-4569-b61d-71fd86b67eba\") "
Jul 14 12:12:28.709928 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:12:28.709815    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b6156d3-9204-4569-b61d-71fd86b67eba-maas-api-tls\") pod \"2b6156d3-9204-4569-b61d-71fd86b67eba\" (UID: \"2b6156d3-9204-4569-b61d-71fd86b67eba\") "
Jul 14 12:12:28.709928 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:12:28.709835    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b6156d3-9204-4569-b61d-71fd86b67eba-openshift-service-ca\") pod \"2b6156d3-9204-4569-b61d-71fd86b67eba\" (UID: \"2b6156d3-9204-4569-b61d-71fd86b67eba\") "
Jul 14 12:12:28.710374 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:12:28.710312    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b6156d3-9204-4569-b61d-71fd86b67eba-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2b6156d3-9204-4569-b61d-71fd86b67eba" (UID: "2b6156d3-9204-4569-b61d-71fd86b67eba"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 12:12:28.712069 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:12:28.712031    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b6156d3-9204-4569-b61d-71fd86b67eba-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2b6156d3-9204-4569-b61d-71fd86b67eba" (UID: "2b6156d3-9204-4569-b61d-71fd86b67eba"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 12:12:28.712181 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:12:28.712041    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b6156d3-9204-4569-b61d-71fd86b67eba-kube-api-access-mhsxd" (OuterVolumeSpecName: "kube-api-access-mhsxd") pod "2b6156d3-9204-4569-b61d-71fd86b67eba" (UID: "2b6156d3-9204-4569-b61d-71fd86b67eba"). InnerVolumeSpecName "kube-api-access-mhsxd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 12:12:28.810917 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:12:28.810880    2551 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b6156d3-9204-4569-b61d-71fd86b67eba-maas-api-tls\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 12:12:28.810917 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:12:28.810912    2551 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b6156d3-9204-4569-b61d-71fd86b67eba-openshift-service-ca\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 12:12:28.810917 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:12:28.810922    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhsxd\" (UniqueName: \"kubernetes.io/projected/2b6156d3-9204-4569-b61d-71fd86b67eba-kube-api-access-mhsxd\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 12:12:29.219325 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:12:29.219296    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cp57b_62c98a11-ece4-4497-99e7-0840a7626a56/ovn-acl-logging/0.log"
Jul 14 12:12:29.226872 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:12:29.226847    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cp57b_62c98a11-ece4-4497-99e7-0840a7626a56/ovn-acl-logging/0.log"
Jul 14 12:12:29.596869 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:12:29.596782    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c52b78c6-596b4b7c66-qfr5r"
Jul 14 12:12:29.596869 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:12:29.596793    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c52b78c6-596b4b7c66-qfr5r" event={"ID":"2b6156d3-9204-4569-b61d-71fd86b67eba","Type":"ContainerDied","Data":"dce2d371e0565c65d978ed4291df857ea1d8e94658206d5379338276b00d7c6a"}
Jul 14 12:12:29.596869 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:12:29.596851    2551 scope.go:117] "RemoveContainer" containerID="942f7b1893821c1130168d5dc9fb7b653994a206b9bfe23734826e85ebfd2f12"
Jul 14 12:12:29.617991 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:12:29.617960    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c52b78c6-596b4b7c66-qfr5r"]
Jul 14 12:12:29.622722 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:12:29.622689    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c52b78c6-596b4b7c66-qfr5r"]
Jul 14 12:12:31.191980 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:12:31.191943    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b6156d3-9204-4569-b61d-71fd86b67eba" path="/var/lib/kubelet/pods/2b6156d3-9204-4569-b61d-71fd86b67eba/volumes"
Jul 14 12:12:31.780416 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:12:31.780377    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 12:12:32.852185 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:12:32.852152    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:12:37.821390 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:12:37.821354    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-c52b78c6/e2e-lifecycle-model-c52b78c6-kserve-5d449b6767-zthkx"]
Jul 14 12:12:37.968877 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:12:37.968839    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:12:52.841463 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:12:52.841420    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-c52b78c6/e2e-lifecycle-model-c52b78c6-kserve-5d449b6767-zthkx" podUID="404635b1-e9f8-44a4-81b9-3861f8e69e2a" containerName="main" containerID="cri-o://8b12dc1dc7f186322c8466585372690709979cb9915ee5d6d7c1f17a6aa12028" gracePeriod=45
Jul 14 12:12:53.093902 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:12:53.093848    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-c52b78c6/e2e-lifecycle-model-c52b78c6-kserve-5d449b6767-zthkx"
Jul 14 12:12:53.226614 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:12:53.226562    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vnhrt\" (UniqueName: \"kubernetes.io/projected/404635b1-e9f8-44a4-81b9-3861f8e69e2a-kube-api-access-vnhrt\") pod \"404635b1-e9f8-44a4-81b9-3861f8e69e2a\" (UID: \"404635b1-e9f8-44a4-81b9-3861f8e69e2a\") "
Jul 14 12:12:53.226786 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:12:53.226626    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/404635b1-e9f8-44a4-81b9-3861f8e69e2a-model-cache\") pod \"404635b1-e9f8-44a4-81b9-3861f8e69e2a\" (UID: \"404635b1-e9f8-44a4-81b9-3861f8e69e2a\") "
Jul 14 12:12:53.226786 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:12:53.226643    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/404635b1-e9f8-44a4-81b9-3861f8e69e2a-home\") pod \"404635b1-e9f8-44a4-81b9-3861f8e69e2a\" (UID: \"404635b1-e9f8-44a4-81b9-3861f8e69e2a\") "
Jul 14 12:12:53.226786 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:12:53.226698    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/404635b1-e9f8-44a4-81b9-3861f8e69e2a-dshm\") pod \"404635b1-e9f8-44a4-81b9-3861f8e69e2a\" (UID: \"404635b1-e9f8-44a4-81b9-3861f8e69e2a\") "
Jul 14 12:12:53.226786 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:12:53.226747    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/404635b1-e9f8-44a4-81b9-3861f8e69e2a-tls-certs\") pod \"404635b1-e9f8-44a4-81b9-3861f8e69e2a\" (UID: \"404635b1-e9f8-44a4-81b9-3861f8e69e2a\") "
Jul 14 12:12:53.226786 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:12:53.226784    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/404635b1-e9f8-44a4-81b9-3861f8e69e2a-tmp-dir\") pod \"404635b1-e9f8-44a4-81b9-3861f8e69e2a\" (UID: \"404635b1-e9f8-44a4-81b9-3861f8e69e2a\") "
Jul 14 12:12:53.227062 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:12:53.226938    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/404635b1-e9f8-44a4-81b9-3861f8e69e2a-home" (OuterVolumeSpecName: "home") pod "404635b1-e9f8-44a4-81b9-3861f8e69e2a" (UID: "404635b1-e9f8-44a4-81b9-3861f8e69e2a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 12:12:53.227062 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:12:53.226957    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/404635b1-e9f8-44a4-81b9-3861f8e69e2a-model-cache" (OuterVolumeSpecName: "model-cache") pod "404635b1-e9f8-44a4-81b9-3861f8e69e2a" (UID: "404635b1-e9f8-44a4-81b9-3861f8e69e2a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 12:12:53.227149 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:12:53.227073    2551 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/404635b1-e9f8-44a4-81b9-3861f8e69e2a-model-cache\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 12:12:53.227149 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:12:53.227093    2551 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/404635b1-e9f8-44a4-81b9-3861f8e69e2a-home\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 12:12:53.227233 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:12:53.227182    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/404635b1-e9f8-44a4-81b9-3861f8e69e2a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "404635b1-e9f8-44a4-81b9-3861f8e69e2a" (UID: "404635b1-e9f8-44a4-81b9-3861f8e69e2a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 12:12:53.228803 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:12:53.228779    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/404635b1-e9f8-44a4-81b9-3861f8e69e2a-dshm" (OuterVolumeSpecName: "dshm") pod "404635b1-e9f8-44a4-81b9-3861f8e69e2a" (UID: "404635b1-e9f8-44a4-81b9-3861f8e69e2a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 12:12:53.228803 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:12:53.228796    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/404635b1-e9f8-44a4-81b9-3861f8e69e2a-kube-api-access-vnhrt" (OuterVolumeSpecName: "kube-api-access-vnhrt") pod "404635b1-e9f8-44a4-81b9-3861f8e69e2a" (UID: "404635b1-e9f8-44a4-81b9-3861f8e69e2a"). InnerVolumeSpecName "kube-api-access-vnhrt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 12:12:53.228959 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:12:53.228841    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/404635b1-e9f8-44a4-81b9-3861f8e69e2a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "404635b1-e9f8-44a4-81b9-3861f8e69e2a" (UID: "404635b1-e9f8-44a4-81b9-3861f8e69e2a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 12:12:53.327908 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:12:53.327878    2551 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/404635b1-e9f8-44a4-81b9-3861f8e69e2a-dshm\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 12:12:53.327908 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:12:53.327903    2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/404635b1-e9f8-44a4-81b9-3861f8e69e2a-tls-certs\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 12:12:53.327908 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:12:53.327913    2551 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/404635b1-e9f8-44a4-81b9-3861f8e69e2a-tmp-dir\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 12:12:53.328125 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:12:53.327922    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnhrt\" (UniqueName: \"kubernetes.io/projected/404635b1-e9f8-44a4-81b9-3861f8e69e2a-kube-api-access-vnhrt\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 12:12:53.707075 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:12:53.707031    2551 generic.go:356] "Generic (PLEG): container finished" podID="404635b1-e9f8-44a4-81b9-3861f8e69e2a" containerID="8b12dc1dc7f186322c8466585372690709979cb9915ee5d6d7c1f17a6aa12028" exitCode=0
Jul 14 12:12:53.707271 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:12:53.707083    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-c52b78c6/e2e-lifecycle-model-c52b78c6-kserve-5d449b6767-zthkx"
Jul 14 12:12:53.707271 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:12:53.707097    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-c52b78c6/e2e-lifecycle-model-c52b78c6-kserve-5d449b6767-zthkx" event={"ID":"404635b1-e9f8-44a4-81b9-3861f8e69e2a","Type":"ContainerDied","Data":"8b12dc1dc7f186322c8466585372690709979cb9915ee5d6d7c1f17a6aa12028"}
Jul 14 12:12:53.707271 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:12:53.707128    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-c52b78c6/e2e-lifecycle-model-c52b78c6-kserve-5d449b6767-zthkx" event={"ID":"404635b1-e9f8-44a4-81b9-3861f8e69e2a","Type":"ContainerDied","Data":"52891c02f81519f83eded0e06a33c021532889b2e089000356fcbd1452a0e318"}
Jul 14 12:12:53.707271 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:12:53.707147    2551 scope.go:117] "RemoveContainer" containerID="8b12dc1dc7f186322c8466585372690709979cb9915ee5d6d7c1f17a6aa12028"
Jul 14 12:12:53.715630 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:12:53.715609    2551 scope.go:117] "RemoveContainer" containerID="8b12dc1dc7f186322c8466585372690709979cb9915ee5d6d7c1f17a6aa12028"
Jul 14 12:12:53.715877 ip-10-0-136-173 kubenswrapper[2551]: E0714 12:12:53.715855    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b12dc1dc7f186322c8466585372690709979cb9915ee5d6d7c1f17a6aa12028\": container with ID starting with 8b12dc1dc7f186322c8466585372690709979cb9915ee5d6d7c1f17a6aa12028 not found: ID does not exist" containerID="8b12dc1dc7f186322c8466585372690709979cb9915ee5d6d7c1f17a6aa12028"
Jul 14 12:12:53.715932 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:12:53.715885    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b12dc1dc7f186322c8466585372690709979cb9915ee5d6d7c1f17a6aa12028"} err="failed to get container status \"8b12dc1dc7f186322c8466585372690709979cb9915ee5d6d7c1f17a6aa12028\": rpc error: code = NotFound desc = could not find container \"8b12dc1dc7f186322c8466585372690709979cb9915ee5d6d7c1f17a6aa12028\": container with ID starting with 8b12dc1dc7f186322c8466585372690709979cb9915ee5d6d7c1f17a6aa12028 not found: ID does not exist"
Jul 14 12:12:53.732121 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:12:53.732093    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-c52b78c6/e2e-lifecycle-model-c52b78c6-kserve-5d449b6767-zthkx"]
Jul 14 12:12:53.737204 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:12:53.737182    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-c52b78c6/e2e-lifecycle-model-c52b78c6-kserve-5d449b6767-zthkx"]
Jul 14 12:12:55.190830 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:12:55.190799    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="404635b1-e9f8-44a4-81b9-3861f8e69e2a" path="/var/lib/kubelet/pods/404635b1-e9f8-44a4-81b9-3861f8e69e2a/volumes"
Jul 14 12:13:14.180008 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:13:14.179976    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:13:16.358697 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:13:16.358666    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:13:23.324161 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:13:23.324073    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:13:23.422979 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:13:23.422944    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:13:35.411330 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:13:35.411288    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:13:35.669942 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:13:35.669836    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:13:36.716267 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:13:36.716238    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:13:36.856836 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:13:36.856802    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:13:36.958802 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:13:36.958767    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:13:37.720880 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:13:37.720843    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:13:37.861896 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:13:37.861863    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:13:38.997401 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:13:38.997361    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 12:13:39.420393 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:13:39.420305    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:13:46.553011 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:13:46.552969    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:13:46.652162 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:13:46.652129    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:13:47.376379 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:13:47.376341    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:13:51.324179 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:13:51.324141    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 12:13:53.333767 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:13:53.333727    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:13:54.234859 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:13:54.234820    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:13:54.339475 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:13:54.339444    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:13:54.434979 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:13:54.434946    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:14:05.048898 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:05.048860    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:14:05.394372 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:05.394279    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:14:26.419018 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:26.418984    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:14:33.048164 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:33.048121    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:14:33.193095 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:33.193060    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:14:33.421881 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:33.421789    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:14:33.802216 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:33.802181    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:14:34.540691 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:34.540652    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:14:36.901738 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:36.901696    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:14:36.980223 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:36.980189    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ae12a0-5b98b64dd4-4bxg7"]
Jul 14 12:14:36.980532 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:36.980518    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="404635b1-e9f8-44a4-81b9-3861f8e69e2a" containerName="main"
Jul 14 12:14:36.980532 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:36.980532    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="404635b1-e9f8-44a4-81b9-3861f8e69e2a" containerName="main"
Jul 14 12:14:36.980700 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:36.980542    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b6156d3-9204-4569-b61d-71fd86b67eba" containerName="maas-api"
Jul 14 12:14:36.980700 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:36.980548    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b6156d3-9204-4569-b61d-71fd86b67eba" containerName="maas-api"
Jul 14 12:14:36.980700 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:36.980628    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b6156d3-9204-4569-b61d-71fd86b67eba" containerName="maas-api"
Jul 14 12:14:36.980700 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:36.980639    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="404635b1-e9f8-44a4-81b9-3861f8e69e2a" containerName="main"
Jul 14 12:14:36.983567 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:36.983546    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae12a0-5b98b64dd4-4bxg7"
Jul 14 12:14:36.986420 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:36.986393    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-62k4x\""
Jul 14 12:14:36.986420 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:36.986411    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 14 12:14:36.987462 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:36.987442    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-ae12a0-serving-cert\""
Jul 14 12:14:36.987462 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:36.987456    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 14 12:14:36.998317 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:36.998293    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ae12a0-5b98b64dd4-4bxg7"]
Jul 14 12:14:37.015140 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:37.015107    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:14:37.116228 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:37.116193    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:14:37.147539 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:37.147498    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6f9bk\" (UniqueName: \"kubernetes.io/projected/40cb117f-688b-4f6f-8597-755788182771-kube-api-access-6f9bk\") pod \"maas-api-e2e-inf-a-ae12a0-5b98b64dd4-4bxg7\" (UID: \"40cb117f-688b-4f6f-8597-755788182771\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae12a0-5b98b64dd4-4bxg7"
Jul 14 12:14:37.147737 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:37.147553    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40cb117f-688b-4f6f-8597-755788182771-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ae12a0-5b98b64dd4-4bxg7\" (UID: \"40cb117f-688b-4f6f-8597-755788182771\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae12a0-5b98b64dd4-4bxg7"
Jul 14 12:14:37.147737 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:37.147698    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40cb117f-688b-4f6f-8597-755788182771-maas-api-tls\") pod \"maas-api-e2e-inf-a-ae12a0-5b98b64dd4-4bxg7\" (UID: \"40cb117f-688b-4f6f-8597-755788182771\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae12a0-5b98b64dd4-4bxg7"
Jul 14 12:14:37.248526 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:37.248491    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6f9bk\" (UniqueName: \"kubernetes.io/projected/40cb117f-688b-4f6f-8597-755788182771-kube-api-access-6f9bk\") pod \"maas-api-e2e-inf-a-ae12a0-5b98b64dd4-4bxg7\" (UID: \"40cb117f-688b-4f6f-8597-755788182771\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae12a0-5b98b64dd4-4bxg7"
Jul 14 12:14:37.248752 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:37.248554    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40cb117f-688b-4f6f-8597-755788182771-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ae12a0-5b98b64dd4-4bxg7\" (UID: \"40cb117f-688b-4f6f-8597-755788182771\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae12a0-5b98b64dd4-4bxg7"
Jul 14 12:14:37.248752 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:37.248642    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40cb117f-688b-4f6f-8597-755788182771-maas-api-tls\") pod \"maas-api-e2e-inf-a-ae12a0-5b98b64dd4-4bxg7\" (UID: \"40cb117f-688b-4f6f-8597-755788182771\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae12a0-5b98b64dd4-4bxg7"
Jul 14 12:14:37.248918 ip-10-0-136-173 kubenswrapper[2551]: E0714 12:14:37.248748    2551 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-ae12a0-serving-cert: secret "maas-api-e2e-inf-a-ae12a0-serving-cert" not found
Jul 14 12:14:37.248918 ip-10-0-136-173 kubenswrapper[2551]: E0714 12:14:37.248815    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40cb117f-688b-4f6f-8597-755788182771-maas-api-tls podName:40cb117f-688b-4f6f-8597-755788182771 nodeName:}" failed. No retries permitted until 2026-07-14 12:14:37.748795103 +0000 UTC m=+2228.980604380 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/40cb117f-688b-4f6f-8597-755788182771-maas-api-tls") pod "maas-api-e2e-inf-a-ae12a0-5b98b64dd4-4bxg7" (UID: "40cb117f-688b-4f6f-8597-755788182771") : secret "maas-api-e2e-inf-a-ae12a0-serving-cert" not found
Jul 14 12:14:37.249352 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:37.249332    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40cb117f-688b-4f6f-8597-755788182771-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ae12a0-5b98b64dd4-4bxg7\" (UID: \"40cb117f-688b-4f6f-8597-755788182771\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae12a0-5b98b64dd4-4bxg7"
Jul 14 12:14:37.265892 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:37.265853    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6f9bk\" (UniqueName: \"kubernetes.io/projected/40cb117f-688b-4f6f-8597-755788182771-kube-api-access-6f9bk\") pod \"maas-api-e2e-inf-a-ae12a0-5b98b64dd4-4bxg7\" (UID: \"40cb117f-688b-4f6f-8597-755788182771\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae12a0-5b98b64dd4-4bxg7"
Jul 14 12:14:37.393962 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:37.393924    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:14:37.754740 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:37.754694    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40cb117f-688b-4f6f-8597-755788182771-maas-api-tls\") pod \"maas-api-e2e-inf-a-ae12a0-5b98b64dd4-4bxg7\" (UID: \"40cb117f-688b-4f6f-8597-755788182771\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae12a0-5b98b64dd4-4bxg7"
Jul 14 12:14:37.757212 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:37.757177    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40cb117f-688b-4f6f-8597-755788182771-maas-api-tls\") pod \"maas-api-e2e-inf-a-ae12a0-5b98b64dd4-4bxg7\" (UID: \"40cb117f-688b-4f6f-8597-755788182771\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae12a0-5b98b64dd4-4bxg7"
Jul 14 12:14:37.859059 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:37.859011    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:14:37.894260 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:37.894218    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae12a0-5b98b64dd4-4bxg7"
Jul 14 12:14:37.959141 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:37.959089    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:14:38.042123 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:38.041936    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ae12a0-5b98b64dd4-4bxg7"]
Jul 14 12:14:38.044843 ip-10-0-136-173 kubenswrapper[2551]: W0714 12:14:38.044804    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40cb117f_688b_4f6f_8597_755788182771.slice/crio-2d6abdf1dd7f8cfaac48431629a333904385d342779c6a7c3318315ef6a825d6 WatchSource:0}: Error finding container 2d6abdf1dd7f8cfaac48431629a333904385d342779c6a7c3318315ef6a825d6: Status 404 returned error can't find the container with id 2d6abdf1dd7f8cfaac48431629a333904385d342779c6a7c3318315ef6a825d6
Jul 14 12:14:38.195876 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:38.195831    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae12a0-5b98b64dd4-4bxg7" event={"ID":"40cb117f-688b-4f6f-8597-755788182771","Type":"ContainerStarted","Data":"2d6abdf1dd7f8cfaac48431629a333904385d342779c6a7c3318315ef6a825d6"}
Jul 14 12:14:39.201597 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:39.201540    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae12a0-5b98b64dd4-4bxg7" event={"ID":"40cb117f-688b-4f6f-8597-755788182771","Type":"ContainerStarted","Data":"e77efb6a2ea32ec7689d2033ec3cfaa27be800afb7519bcabaaf5ea9b3f94cc1"}
Jul 14 12:14:39.202049 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:39.201625    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae12a0-5b98b64dd4-4bxg7"
Jul 14 12:14:39.222181 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:39.222126    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae12a0-5b98b64dd4-4bxg7" podStartSLOduration=2.7641708019999998 podStartE2EDuration="3.222110029s" totalImagesPullingTime="457.939227ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 12:14:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 12:14:39.221210017 +0000 UTC m=+2230.453019318" watchObservedRunningTime="2026-07-14 12:14:39.222110029 +0000 UTC m=+2230.453919324"
Jul 14 12:14:41.543619 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:41.543570    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:14:41.877818 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:41.877733    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:14:42.221420 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:42.221380    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:14:42.335617 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:42.335541    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:14:44.585717 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:44.585674    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 12:14:44.941337 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:44.941302    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:14:45.025799 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:45.025762    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a015ba-69c489d9f7-ffwkz"]
Jul 14 12:14:45.029442 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:45.029415    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a015ba-69c489d9f7-ffwkz"
Jul 14 12:14:45.032114 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:45.032091    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-a015ba-serving-cert\""
Jul 14 12:14:45.041248 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:45.041220    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a015ba-69c489d9f7-ffwkz"]
Jul 14 12:14:45.065838 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:45.065806    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:14:45.115909 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:45.115871    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7srd2\" (UniqueName: \"kubernetes.io/projected/eae5d888-02f4-462c-99f4-3c9a805ff627-kube-api-access-7srd2\") pod \"maas-api-e2e-inf-b-a015ba-69c489d9f7-ffwkz\" (UID: \"eae5d888-02f4-462c-99f4-3c9a805ff627\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a015ba-69c489d9f7-ffwkz"
Jul 14 12:14:45.116144 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:45.115916    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eae5d888-02f4-462c-99f4-3c9a805ff627-maas-api-tls\") pod \"maas-api-e2e-inf-b-a015ba-69c489d9f7-ffwkz\" (UID: \"eae5d888-02f4-462c-99f4-3c9a805ff627\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a015ba-69c489d9f7-ffwkz"
Jul 14 12:14:45.116144 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:45.116001    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eae5d888-02f4-462c-99f4-3c9a805ff627-openshift-service-ca\") pod \"maas-api-e2e-inf-b-a015ba-69c489d9f7-ffwkz\" (UID: \"eae5d888-02f4-462c-99f4-3c9a805ff627\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a015ba-69c489d9f7-ffwkz"
Jul 14 12:14:45.206349 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:45.206261    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:14:45.212087 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:45.212059    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae12a0-5b98b64dd4-4bxg7"
Jul 14 12:14:45.216826 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:45.216802    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eae5d888-02f4-462c-99f4-3c9a805ff627-openshift-service-ca\") pod \"maas-api-e2e-inf-b-a015ba-69c489d9f7-ffwkz\" (UID: \"eae5d888-02f4-462c-99f4-3c9a805ff627\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a015ba-69c489d9f7-ffwkz"
Jul 14 12:14:45.216942 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:45.216906    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7srd2\" (UniqueName: \"kubernetes.io/projected/eae5d888-02f4-462c-99f4-3c9a805ff627-kube-api-access-7srd2\") pod \"maas-api-e2e-inf-b-a015ba-69c489d9f7-ffwkz\" (UID: \"eae5d888-02f4-462c-99f4-3c9a805ff627\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a015ba-69c489d9f7-ffwkz"
Jul 14 12:14:45.217003 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:45.216964    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eae5d888-02f4-462c-99f4-3c9a805ff627-maas-api-tls\") pod \"maas-api-e2e-inf-b-a015ba-69c489d9f7-ffwkz\" (UID: \"eae5d888-02f4-462c-99f4-3c9a805ff627\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a015ba-69c489d9f7-ffwkz"
Jul 14 12:14:45.217168 ip-10-0-136-173 kubenswrapper[2551]: E0714 12:14:45.217143    2551 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-a015ba-serving-cert: secret "maas-api-e2e-inf-b-a015ba-serving-cert" not found
Jul 14 12:14:45.217249 ip-10-0-136-173 kubenswrapper[2551]: E0714 12:14:45.217238    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eae5d888-02f4-462c-99f4-3c9a805ff627-maas-api-tls podName:eae5d888-02f4-462c-99f4-3c9a805ff627 nodeName:}" failed. No retries permitted until 2026-07-14 12:14:45.717214455 +0000 UTC m=+2236.949023740 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/eae5d888-02f4-462c-99f4-3c9a805ff627-maas-api-tls") pod "maas-api-e2e-inf-b-a015ba-69c489d9f7-ffwkz" (UID: "eae5d888-02f4-462c-99f4-3c9a805ff627") : secret "maas-api-e2e-inf-b-a015ba-serving-cert" not found
Jul 14 12:14:45.217481 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:45.217462    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eae5d888-02f4-462c-99f4-3c9a805ff627-openshift-service-ca\") pod \"maas-api-e2e-inf-b-a015ba-69c489d9f7-ffwkz\" (UID: \"eae5d888-02f4-462c-99f4-3c9a805ff627\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a015ba-69c489d9f7-ffwkz"
Jul 14 12:14:45.235308 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:45.235277    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7srd2\" (UniqueName: \"kubernetes.io/projected/eae5d888-02f4-462c-99f4-3c9a805ff627-kube-api-access-7srd2\") pod \"maas-api-e2e-inf-b-a015ba-69c489d9f7-ffwkz\" (UID: \"eae5d888-02f4-462c-99f4-3c9a805ff627\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a015ba-69c489d9f7-ffwkz"
Jul 14 12:14:45.536437 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:45.536352    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:14:45.721768 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:45.721713    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eae5d888-02f4-462c-99f4-3c9a805ff627-maas-api-tls\") pod \"maas-api-e2e-inf-b-a015ba-69c489d9f7-ffwkz\" (UID: \"eae5d888-02f4-462c-99f4-3c9a805ff627\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a015ba-69c489d9f7-ffwkz"
Jul 14 12:14:45.724537 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:45.724501    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eae5d888-02f4-462c-99f4-3c9a805ff627-maas-api-tls\") pod \"maas-api-e2e-inf-b-a015ba-69c489d9f7-ffwkz\" (UID: \"eae5d888-02f4-462c-99f4-3c9a805ff627\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a015ba-69c489d9f7-ffwkz"
Jul 14 12:14:45.932459 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:45.932355    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:14:45.941213 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:45.941180    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a015ba-69c489d9f7-ffwkz"
Jul 14 12:14:46.093915 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:46.093839    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a015ba-69c489d9f7-ffwkz"]
Jul 14 12:14:46.096883 ip-10-0-136-173 kubenswrapper[2551]: W0714 12:14:46.096852    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeae5d888_02f4_462c_99f4_3c9a805ff627.slice/crio-5f18d5c41dd66490ad8f8c744830633e6fb89ae10c30bf61fbf40c2cf252325e WatchSource:0}: Error finding container 5f18d5c41dd66490ad8f8c744830633e6fb89ae10c30bf61fbf40c2cf252325e: Status 404 returned error can't find the container with id 5f18d5c41dd66490ad8f8c744830633e6fb89ae10c30bf61fbf40c2cf252325e
Jul 14 12:14:46.151734 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:46.151695    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:14:46.239791 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:46.239751    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a015ba-69c489d9f7-ffwkz" event={"ID":"eae5d888-02f4-462c-99f4-3c9a805ff627","Type":"ContainerStarted","Data":"5f18d5c41dd66490ad8f8c744830633e6fb89ae10c30bf61fbf40c2cf252325e"}
Jul 14 12:14:47.246195 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:47.246156    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a015ba-69c489d9f7-ffwkz" event={"ID":"eae5d888-02f4-462c-99f4-3c9a805ff627","Type":"ContainerStarted","Data":"ad92042d522160e73a2aa501c672b4147bf8af663c2d729a30cd9512615e5038"}
Jul 14 12:14:47.246650 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:47.246272    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a015ba-69c489d9f7-ffwkz"
Jul 14 12:14:47.289213 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:47.289160    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a015ba-69c489d9f7-ffwkz" podStartSLOduration=1.9723289560000001 podStartE2EDuration="2.28914437s" totalImagesPullingTime="316.815414ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 12:14:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 12:14:47.288819535 +0000 UTC m=+2238.520628853" watchObservedRunningTime="2026-07-14 12:14:47.28914437 +0000 UTC m=+2238.520953666"
Jul 14 12:14:49.456793 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:49.456752    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:14:53.256672 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:53.256592    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a015ba-69c489d9f7-ffwkz"
Jul 14 12:14:53.796658 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:53.796621    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:14:54.477927 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:54.477889    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-ae12a0/test-model-ae12a0-kserve-745db7bdbf-2wzhl"]
Jul 14 12:14:54.481385 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:54.481364    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ae12a0/test-model-ae12a0-kserve-745db7bdbf-2wzhl"
Jul 14 12:14:54.484919 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:54.484891    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-ae12a0\"/\"openshift-service-ca.crt\""
Jul 14 12:14:54.486281 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:54.486257    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-ae12a0\"/\"test-model-ae12a0-kserve-self-signed-certs\""
Jul 14 12:14:54.486409 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:54.486262    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-ae12a0\"/\"kube-root-ca.crt\""
Jul 14 12:14:54.486409 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:54.486305    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-ae12a0\"/\"default-dockercfg-whwpl\""
Jul 14 12:14:54.495191 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:54.495161    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-ae12a0/test-model-ae12a0-kserve-745db7bdbf-2wzhl"]
Jul 14 12:14:54.522023 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:54.521986    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:14:54.600629 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:54.600563    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdfvb\" (UniqueName: \"kubernetes.io/projected/671c4f06-dc99-4c32-a2e9-21a49361ae47-kube-api-access-vdfvb\") pod \"test-model-ae12a0-kserve-745db7bdbf-2wzhl\" (UID: \"671c4f06-dc99-4c32-a2e9-21a49361ae47\") " pod="ai-tenant-e2e-inf-a-ae12a0/test-model-ae12a0-kserve-745db7bdbf-2wzhl"
Jul 14 12:14:54.600788 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:54.600710    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/671c4f06-dc99-4c32-a2e9-21a49361ae47-model-cache\") pod \"test-model-ae12a0-kserve-745db7bdbf-2wzhl\" (UID: \"671c4f06-dc99-4c32-a2e9-21a49361ae47\") " pod="ai-tenant-e2e-inf-a-ae12a0/test-model-ae12a0-kserve-745db7bdbf-2wzhl"
Jul 14 12:14:54.600788 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:54.600740    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/671c4f06-dc99-4c32-a2e9-21a49361ae47-tls-certs\") pod \"test-model-ae12a0-kserve-745db7bdbf-2wzhl\" (UID: \"671c4f06-dc99-4c32-a2e9-21a49361ae47\") " pod="ai-tenant-e2e-inf-a-ae12a0/test-model-ae12a0-kserve-745db7bdbf-2wzhl"
Jul 14 12:14:54.600788 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:54.600768    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/671c4f06-dc99-4c32-a2e9-21a49361ae47-home\") pod \"test-model-ae12a0-kserve-745db7bdbf-2wzhl\" (UID: \"671c4f06-dc99-4c32-a2e9-21a49361ae47\") " pod="ai-tenant-e2e-inf-a-ae12a0/test-model-ae12a0-kserve-745db7bdbf-2wzhl"
Jul 14 12:14:54.600915 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:54.600821    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/671c4f06-dc99-4c32-a2e9-21a49361ae47-tmp-dir\") pod \"test-model-ae12a0-kserve-745db7bdbf-2wzhl\" (UID: \"671c4f06-dc99-4c32-a2e9-21a49361ae47\") " pod="ai-tenant-e2e-inf-a-ae12a0/test-model-ae12a0-kserve-745db7bdbf-2wzhl"
Jul 14 12:14:54.600915 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:54.600854    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/671c4f06-dc99-4c32-a2e9-21a49361ae47-dshm\") pod \"test-model-ae12a0-kserve-745db7bdbf-2wzhl\" (UID: \"671c4f06-dc99-4c32-a2e9-21a49361ae47\") " pod="ai-tenant-e2e-inf-a-ae12a0/test-model-ae12a0-kserve-745db7bdbf-2wzhl"
Jul 14 12:14:54.702071 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:54.702024    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdfvb\" (UniqueName: \"kubernetes.io/projected/671c4f06-dc99-4c32-a2e9-21a49361ae47-kube-api-access-vdfvb\") pod \"test-model-ae12a0-kserve-745db7bdbf-2wzhl\" (UID: \"671c4f06-dc99-4c32-a2e9-21a49361ae47\") " pod="ai-tenant-e2e-inf-a-ae12a0/test-model-ae12a0-kserve-745db7bdbf-2wzhl"
Jul 14 12:14:54.702286 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:54.702119    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/671c4f06-dc99-4c32-a2e9-21a49361ae47-model-cache\") pod \"test-model-ae12a0-kserve-745db7bdbf-2wzhl\" (UID: \"671c4f06-dc99-4c32-a2e9-21a49361ae47\") " pod="ai-tenant-e2e-inf-a-ae12a0/test-model-ae12a0-kserve-745db7bdbf-2wzhl"
Jul 14 12:14:54.702286 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:54.702146    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/671c4f06-dc99-4c32-a2e9-21a49361ae47-tls-certs\") pod \"test-model-ae12a0-kserve-745db7bdbf-2wzhl\" (UID: \"671c4f06-dc99-4c32-a2e9-21a49361ae47\") " pod="ai-tenant-e2e-inf-a-ae12a0/test-model-ae12a0-kserve-745db7bdbf-2wzhl"
Jul 14 12:14:54.702286 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:54.702178    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/671c4f06-dc99-4c32-a2e9-21a49361ae47-home\") pod \"test-model-ae12a0-kserve-745db7bdbf-2wzhl\" (UID: \"671c4f06-dc99-4c32-a2e9-21a49361ae47\") " pod="ai-tenant-e2e-inf-a-ae12a0/test-model-ae12a0-kserve-745db7bdbf-2wzhl"
Jul 14 12:14:54.702286 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:54.702206    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/671c4f06-dc99-4c32-a2e9-21a49361ae47-tmp-dir\") pod \"test-model-ae12a0-kserve-745db7bdbf-2wzhl\" (UID: \"671c4f06-dc99-4c32-a2e9-21a49361ae47\") " pod="ai-tenant-e2e-inf-a-ae12a0/test-model-ae12a0-kserve-745db7bdbf-2wzhl"
Jul 14 12:14:54.702286 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:54.702246    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/671c4f06-dc99-4c32-a2e9-21a49361ae47-dshm\") pod \"test-model-ae12a0-kserve-745db7bdbf-2wzhl\" (UID: \"671c4f06-dc99-4c32-a2e9-21a49361ae47\") " pod="ai-tenant-e2e-inf-a-ae12a0/test-model-ae12a0-kserve-745db7bdbf-2wzhl"
Jul 14 12:14:54.702748 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:54.702720    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/671c4f06-dc99-4c32-a2e9-21a49361ae47-model-cache\") pod \"test-model-ae12a0-kserve-745db7bdbf-2wzhl\" (UID: \"671c4f06-dc99-4c32-a2e9-21a49361ae47\") " pod="ai-tenant-e2e-inf-a-ae12a0/test-model-ae12a0-kserve-745db7bdbf-2wzhl"
Jul 14 12:14:54.702748 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:54.702740    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/671c4f06-dc99-4c32-a2e9-21a49361ae47-home\") pod \"test-model-ae12a0-kserve-745db7bdbf-2wzhl\" (UID: \"671c4f06-dc99-4c32-a2e9-21a49361ae47\") " pod="ai-tenant-e2e-inf-a-ae12a0/test-model-ae12a0-kserve-745db7bdbf-2wzhl"
Jul 14 12:14:54.702875 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:54.702821    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/671c4f06-dc99-4c32-a2e9-21a49361ae47-tmp-dir\") pod \"test-model-ae12a0-kserve-745db7bdbf-2wzhl\" (UID: \"671c4f06-dc99-4c32-a2e9-21a49361ae47\") " pod="ai-tenant-e2e-inf-a-ae12a0/test-model-ae12a0-kserve-745db7bdbf-2wzhl"
Jul 14 12:14:54.704623 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:54.704570    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/671c4f06-dc99-4c32-a2e9-21a49361ae47-dshm\") pod \"test-model-ae12a0-kserve-745db7bdbf-2wzhl\" (UID: \"671c4f06-dc99-4c32-a2e9-21a49361ae47\") " pod="ai-tenant-e2e-inf-a-ae12a0/test-model-ae12a0-kserve-745db7bdbf-2wzhl"
Jul 14 12:14:54.704799 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:54.704780    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/671c4f06-dc99-4c32-a2e9-21a49361ae47-tls-certs\") pod \"test-model-ae12a0-kserve-745db7bdbf-2wzhl\" (UID: \"671c4f06-dc99-4c32-a2e9-21a49361ae47\") " pod="ai-tenant-e2e-inf-a-ae12a0/test-model-ae12a0-kserve-745db7bdbf-2wzhl"
Jul 14 12:14:54.713559 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:54.713532    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdfvb\" (UniqueName: \"kubernetes.io/projected/671c4f06-dc99-4c32-a2e9-21a49361ae47-kube-api-access-vdfvb\") pod \"test-model-ae12a0-kserve-745db7bdbf-2wzhl\" (UID: \"671c4f06-dc99-4c32-a2e9-21a49361ae47\") " pod="ai-tenant-e2e-inf-a-ae12a0/test-model-ae12a0-kserve-745db7bdbf-2wzhl"
Jul 14 12:14:54.790995 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:54.790908    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ae12a0/test-model-ae12a0-kserve-745db7bdbf-2wzhl"
Jul 14 12:14:54.938104 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:54.938043    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-ae12a0/test-model-ae12a0-kserve-745db7bdbf-2wzhl"]
Jul 14 12:14:54.940428 ip-10-0-136-173 kubenswrapper[2551]: W0714 12:14:54.940400    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod671c4f06_dc99_4c32_a2e9_21a49361ae47.slice/crio-a3a210522184e61d80d87f276f48267ec9117577dcee53463e9685a986ef5e81 WatchSource:0}: Error finding container a3a210522184e61d80d87f276f48267ec9117577dcee53463e9685a986ef5e81: Status 404 returned error can't find the container with id a3a210522184e61d80d87f276f48267ec9117577dcee53463e9685a986ef5e81
Jul 14 12:14:55.289950 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:55.289903    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ae12a0/test-model-ae12a0-kserve-745db7bdbf-2wzhl" event={"ID":"671c4f06-dc99-4c32-a2e9-21a49361ae47","Type":"ContainerStarted","Data":"f794a7e0d219d7f36a6f2fc82b2f1d01528ac791a0d06a867d5a71c8d9bc686d"}
Jul 14 12:14:55.290137 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:55.289959    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ae12a0/test-model-ae12a0-kserve-745db7bdbf-2wzhl" event={"ID":"671c4f06-dc99-4c32-a2e9-21a49361ae47","Type":"ContainerStarted","Data":"a3a210522184e61d80d87f276f48267ec9117577dcee53463e9685a986ef5e81"}
Jul 14 12:14:55.319181 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:55.319125    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-ae12a0/test-model-ae12a0-kserve-745db7bdbf-2wzhl" podStartSLOduration=1.31910927 podStartE2EDuration="1.31910927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 12:14:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 12:14:55.317937567 +0000 UTC m=+2246.549746877" watchObservedRunningTime="2026-07-14 12:14:55.31910927 +0000 UTC m=+2246.550918565"
Jul 14 12:14:56.091593 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:56.091532    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 12:14:58.690975 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:58.690944    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:14:58.792150 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:14:58.792117    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:15:04.791433 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:04.791396    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-ae12a0/test-model-ae12a0-kserve-745db7bdbf-2wzhl"
Jul 14 12:15:04.791877 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:04.791479    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-ae12a0/test-model-ae12a0-kserve-745db7bdbf-2wzhl"
Jul 14 12:15:04.803697 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:04.803665    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-ae12a0/test-model-ae12a0-kserve-745db7bdbf-2wzhl"
Jul 14 12:15:05.349705 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:05.349674    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-ae12a0/test-model-ae12a0-kserve-745db7bdbf-2wzhl"
Jul 14 12:15:06.935841 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:06.935793    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:15:06.988213 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:06.988166    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 12:15:07.033721 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:07.033683    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:15:22.211709 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:22.211672    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-a015ba/test-model-a015ba-kserve-5ddf59d5-8f5zb"]
Jul 14 12:15:22.214899 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:22.214882    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a015ba/test-model-a015ba-kserve-5ddf59d5-8f5zb"
Jul 14 12:15:22.218728 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:22.218705    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-a015ba\"/\"test-model-a015ba-kserve-self-signed-certs\""
Jul 14 12:15:22.219839 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:22.219815    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-a015ba\"/\"default-dockercfg-bslt2\""
Jul 14 12:15:22.219962 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:22.219817    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-a015ba\"/\"kube-root-ca.crt\""
Jul 14 12:15:22.219962 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:22.219816    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-a015ba\"/\"openshift-service-ca.crt\""
Jul 14 12:15:22.233536 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:22.233506    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-a015ba/test-model-a015ba-kserve-5ddf59d5-8f5zb"]
Jul 14 12:15:22.248197 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:22.248167    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:15:22.370558 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:22.370508    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3e97ba9-2c8b-41c8-813f-72eb1fee24e1-tmp-dir\") pod \"test-model-a015ba-kserve-5ddf59d5-8f5zb\" (UID: \"e3e97ba9-2c8b-41c8-813f-72eb1fee24e1\") " pod="ai-tenant-e2e-inf-b-a015ba/test-model-a015ba-kserve-5ddf59d5-8f5zb"
Jul 14 12:15:22.370558 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:22.370560    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3e97ba9-2c8b-41c8-813f-72eb1fee24e1-tls-certs\") pod \"test-model-a015ba-kserve-5ddf59d5-8f5zb\" (UID: \"e3e97ba9-2c8b-41c8-813f-72eb1fee24e1\") " pod="ai-tenant-e2e-inf-b-a015ba/test-model-a015ba-kserve-5ddf59d5-8f5zb"
Jul 14 12:15:22.370863 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:22.370626    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3e97ba9-2c8b-41c8-813f-72eb1fee24e1-model-cache\") pod \"test-model-a015ba-kserve-5ddf59d5-8f5zb\" (UID: \"e3e97ba9-2c8b-41c8-813f-72eb1fee24e1\") " pod="ai-tenant-e2e-inf-b-a015ba/test-model-a015ba-kserve-5ddf59d5-8f5zb"
Jul 14 12:15:22.370863 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:22.370711    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3e97ba9-2c8b-41c8-813f-72eb1fee24e1-home\") pod \"test-model-a015ba-kserve-5ddf59d5-8f5zb\" (UID: \"e3e97ba9-2c8b-41c8-813f-72eb1fee24e1\") " pod="ai-tenant-e2e-inf-b-a015ba/test-model-a015ba-kserve-5ddf59d5-8f5zb"
Jul 14 12:15:22.370863 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:22.370779    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3e97ba9-2c8b-41c8-813f-72eb1fee24e1-dshm\") pod \"test-model-a015ba-kserve-5ddf59d5-8f5zb\" (UID: \"e3e97ba9-2c8b-41c8-813f-72eb1fee24e1\") " pod="ai-tenant-e2e-inf-b-a015ba/test-model-a015ba-kserve-5ddf59d5-8f5zb"
Jul 14 12:15:22.371011 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:22.370875    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ss756\" (UniqueName: \"kubernetes.io/projected/e3e97ba9-2c8b-41c8-813f-72eb1fee24e1-kube-api-access-ss756\") pod \"test-model-a015ba-kserve-5ddf59d5-8f5zb\" (UID: \"e3e97ba9-2c8b-41c8-813f-72eb1fee24e1\") " pod="ai-tenant-e2e-inf-b-a015ba/test-model-a015ba-kserve-5ddf59d5-8f5zb"
Jul 14 12:15:22.472269 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:22.472167    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ss756\" (UniqueName: \"kubernetes.io/projected/e3e97ba9-2c8b-41c8-813f-72eb1fee24e1-kube-api-access-ss756\") pod \"test-model-a015ba-kserve-5ddf59d5-8f5zb\" (UID: \"e3e97ba9-2c8b-41c8-813f-72eb1fee24e1\") " pod="ai-tenant-e2e-inf-b-a015ba/test-model-a015ba-kserve-5ddf59d5-8f5zb"
Jul 14 12:15:22.472269 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:22.472242    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3e97ba9-2c8b-41c8-813f-72eb1fee24e1-tmp-dir\") pod \"test-model-a015ba-kserve-5ddf59d5-8f5zb\" (UID: \"e3e97ba9-2c8b-41c8-813f-72eb1fee24e1\") " pod="ai-tenant-e2e-inf-b-a015ba/test-model-a015ba-kserve-5ddf59d5-8f5zb"
Jul 14 12:15:22.472490 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:22.472269    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3e97ba9-2c8b-41c8-813f-72eb1fee24e1-tls-certs\") pod \"test-model-a015ba-kserve-5ddf59d5-8f5zb\" (UID: \"e3e97ba9-2c8b-41c8-813f-72eb1fee24e1\") " pod="ai-tenant-e2e-inf-b-a015ba/test-model-a015ba-kserve-5ddf59d5-8f5zb"
Jul 14 12:15:22.472490 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:22.472458    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3e97ba9-2c8b-41c8-813f-72eb1fee24e1-model-cache\") pod \"test-model-a015ba-kserve-5ddf59d5-8f5zb\" (UID: \"e3e97ba9-2c8b-41c8-813f-72eb1fee24e1\") " pod="ai-tenant-e2e-inf-b-a015ba/test-model-a015ba-kserve-5ddf59d5-8f5zb"
Jul 14 12:15:22.473038 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:22.472526    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3e97ba9-2c8b-41c8-813f-72eb1fee24e1-home\") pod \"test-model-a015ba-kserve-5ddf59d5-8f5zb\" (UID: \"e3e97ba9-2c8b-41c8-813f-72eb1fee24e1\") " pod="ai-tenant-e2e-inf-b-a015ba/test-model-a015ba-kserve-5ddf59d5-8f5zb"
Jul 14 12:15:22.473038 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:22.472591    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3e97ba9-2c8b-41c8-813f-72eb1fee24e1-dshm\") pod \"test-model-a015ba-kserve-5ddf59d5-8f5zb\" (UID: \"e3e97ba9-2c8b-41c8-813f-72eb1fee24e1\") " pod="ai-tenant-e2e-inf-b-a015ba/test-model-a015ba-kserve-5ddf59d5-8f5zb"
Jul 14 12:15:22.473038 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:22.472684    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3e97ba9-2c8b-41c8-813f-72eb1fee24e1-tmp-dir\") pod \"test-model-a015ba-kserve-5ddf59d5-8f5zb\" (UID: \"e3e97ba9-2c8b-41c8-813f-72eb1fee24e1\") " pod="ai-tenant-e2e-inf-b-a015ba/test-model-a015ba-kserve-5ddf59d5-8f5zb"
Jul 14 12:15:22.473038 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:22.472840    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3e97ba9-2c8b-41c8-813f-72eb1fee24e1-model-cache\") pod \"test-model-a015ba-kserve-5ddf59d5-8f5zb\" (UID: \"e3e97ba9-2c8b-41c8-813f-72eb1fee24e1\") " pod="ai-tenant-e2e-inf-b-a015ba/test-model-a015ba-kserve-5ddf59d5-8f5zb"
Jul 14 12:15:22.473038 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:22.472860    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3e97ba9-2c8b-41c8-813f-72eb1fee24e1-home\") pod \"test-model-a015ba-kserve-5ddf59d5-8f5zb\" (UID: \"e3e97ba9-2c8b-41c8-813f-72eb1fee24e1\") " pod="ai-tenant-e2e-inf-b-a015ba/test-model-a015ba-kserve-5ddf59d5-8f5zb"
Jul 14 12:15:22.475125 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:22.475103    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3e97ba9-2c8b-41c8-813f-72eb1fee24e1-tls-certs\") pod \"test-model-a015ba-kserve-5ddf59d5-8f5zb\" (UID: \"e3e97ba9-2c8b-41c8-813f-72eb1fee24e1\") " pod="ai-tenant-e2e-inf-b-a015ba/test-model-a015ba-kserve-5ddf59d5-8f5zb"
Jul 14 12:15:22.475261 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:22.475250    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3e97ba9-2c8b-41c8-813f-72eb1fee24e1-dshm\") pod \"test-model-a015ba-kserve-5ddf59d5-8f5zb\" (UID: \"e3e97ba9-2c8b-41c8-813f-72eb1fee24e1\") " pod="ai-tenant-e2e-inf-b-a015ba/test-model-a015ba-kserve-5ddf59d5-8f5zb"
Jul 14 12:15:22.481893 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:22.481866    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ss756\" (UniqueName: \"kubernetes.io/projected/e3e97ba9-2c8b-41c8-813f-72eb1fee24e1-kube-api-access-ss756\") pod \"test-model-a015ba-kserve-5ddf59d5-8f5zb\" (UID: \"e3e97ba9-2c8b-41c8-813f-72eb1fee24e1\") " pod="ai-tenant-e2e-inf-b-a015ba/test-model-a015ba-kserve-5ddf59d5-8f5zb"
Jul 14 12:15:22.525494 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:22.525455    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a015ba/test-model-a015ba-kserve-5ddf59d5-8f5zb"
Jul 14 12:15:22.673102 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:22.673067    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-a015ba/test-model-a015ba-kserve-5ddf59d5-8f5zb"]
Jul 14 12:15:22.676128 ip-10-0-136-173 kubenswrapper[2551]: W0714 12:15:22.676095    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode3e97ba9_2c8b_41c8_813f_72eb1fee24e1.slice/crio-62eea4cf6c00236edf5c4dd9d36f684906531bc8e451b506e912328ef449e8e7 WatchSource:0}: Error finding container 62eea4cf6c00236edf5c4dd9d36f684906531bc8e451b506e912328ef449e8e7: Status 404 returned error can't find the container with id 62eea4cf6c00236edf5c4dd9d36f684906531bc8e451b506e912328ef449e8e7
Jul 14 12:15:23.422914 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:23.422878    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a015ba/test-model-a015ba-kserve-5ddf59d5-8f5zb" event={"ID":"e3e97ba9-2c8b-41c8-813f-72eb1fee24e1","Type":"ContainerStarted","Data":"d8c8f44b8626c2507f5178f99c1fbeff281149c4ed8fdfd86f8d9c4e7fb54a6c"}
Jul 14 12:15:23.423340 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:23.422926    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a015ba/test-model-a015ba-kserve-5ddf59d5-8f5zb" event={"ID":"e3e97ba9-2c8b-41c8-813f-72eb1fee24e1","Type":"ContainerStarted","Data":"62eea4cf6c00236edf5c4dd9d36f684906531bc8e451b506e912328ef449e8e7"}
Jul 14 12:15:23.452185 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:23.452131    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-a015ba/test-model-a015ba-kserve-5ddf59d5-8f5zb" podStartSLOduration=1.452113839 podStartE2EDuration="1.452113839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 12:15:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 12:15:23.450302079 +0000 UTC m=+2274.682111393" watchObservedRunningTime="2026-07-14 12:15:23.452113839 +0000 UTC m=+2274.683923176"
Jul 14 12:15:27.088484 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:27.088447    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 12:15:32.525663 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:32.525616    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-a015ba/test-model-a015ba-kserve-5ddf59d5-8f5zb"
Jul 14 12:15:32.526306 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:32.525689    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-a015ba/test-model-a015ba-kserve-5ddf59d5-8f5zb"
Jul 14 12:15:32.538334 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:32.538299    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-a015ba/test-model-a015ba-kserve-5ddf59d5-8f5zb"
Jul 14 12:15:33.483118 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:33.483086    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-a015ba/test-model-a015ba-kserve-5ddf59d5-8f5zb"
Jul 14 12:15:42.891269 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:42.891234    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 12:15:42.943703 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:42.943669    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-ae12a0/test-model-ae12a0-kserve-745db7bdbf-2wzhl"]
Jul 14 12:15:42.981207 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:42.981175    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:15:43.908666 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:43.908630    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-a015ba/test-model-a015ba-kserve-5ddf59d5-8f5zb"]
Jul 14 12:15:43.946253 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:43.946203    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:15:44.732934 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:44.732890    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:15:44.857007 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:44.856972    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:15:54.494831 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:54.494792    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ae12a0-5b98b64dd4-4bxg7"]
Jul 14 12:15:54.495273 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:54.495029    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae12a0-5b98b64dd4-4bxg7" podUID="40cb117f-688b-4f6f-8597-755788182771" containerName="maas-api" containerID="cri-o://e77efb6a2ea32ec7689d2033ec3cfaa27be800afb7519bcabaaf5ea9b3f94cc1" gracePeriod=30
Jul 14 12:15:54.539946 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:54.539911    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:15:54.755345 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:54.755280    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae12a0-5b98b64dd4-4bxg7"
Jul 14 12:15:54.887740 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:54.887702    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6f9bk\" (UniqueName: \"kubernetes.io/projected/40cb117f-688b-4f6f-8597-755788182771-kube-api-access-6f9bk\") pod \"40cb117f-688b-4f6f-8597-755788182771\" (UID: \"40cb117f-688b-4f6f-8597-755788182771\") "
Jul 14 12:15:54.887927 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:54.887796    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40cb117f-688b-4f6f-8597-755788182771-maas-api-tls\") pod \"40cb117f-688b-4f6f-8597-755788182771\" (UID: \"40cb117f-688b-4f6f-8597-755788182771\") "
Jul 14 12:15:54.887927 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:54.887865    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40cb117f-688b-4f6f-8597-755788182771-openshift-service-ca\") pod \"40cb117f-688b-4f6f-8597-755788182771\" (UID: \"40cb117f-688b-4f6f-8597-755788182771\") "
Jul 14 12:15:54.888268 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:54.888242    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40cb117f-688b-4f6f-8597-755788182771-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "40cb117f-688b-4f6f-8597-755788182771" (UID: "40cb117f-688b-4f6f-8597-755788182771"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 12:15:54.890017 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:54.889993    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40cb117f-688b-4f6f-8597-755788182771-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "40cb117f-688b-4f6f-8597-755788182771" (UID: "40cb117f-688b-4f6f-8597-755788182771"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 12:15:54.890143 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:54.890118    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40cb117f-688b-4f6f-8597-755788182771-kube-api-access-6f9bk" (OuterVolumeSpecName: "kube-api-access-6f9bk") pod "40cb117f-688b-4f6f-8597-755788182771" (UID: "40cb117f-688b-4f6f-8597-755788182771"). InnerVolumeSpecName "kube-api-access-6f9bk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 12:15:54.988681 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:54.988628    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6f9bk\" (UniqueName: \"kubernetes.io/projected/40cb117f-688b-4f6f-8597-755788182771-kube-api-access-6f9bk\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 12:15:54.988681 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:54.988675    2551 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40cb117f-688b-4f6f-8597-755788182771-maas-api-tls\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 12:15:54.988681 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:54.988686    2551 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40cb117f-688b-4f6f-8597-755788182771-openshift-service-ca\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 12:15:55.397204 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:55.397166    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 12:15:55.582534 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:55.582432    2551 generic.go:356] "Generic (PLEG): container finished" podID="40cb117f-688b-4f6f-8597-755788182771" containerID="e77efb6a2ea32ec7689d2033ec3cfaa27be800afb7519bcabaaf5ea9b3f94cc1" exitCode=0
Jul 14 12:15:55.582534 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:55.582476    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae12a0-5b98b64dd4-4bxg7"
Jul 14 12:15:55.582534 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:55.582520    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae12a0-5b98b64dd4-4bxg7" event={"ID":"40cb117f-688b-4f6f-8597-755788182771","Type":"ContainerDied","Data":"e77efb6a2ea32ec7689d2033ec3cfaa27be800afb7519bcabaaf5ea9b3f94cc1"}
Jul 14 12:15:55.583095 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:55.582606    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae12a0-5b98b64dd4-4bxg7" event={"ID":"40cb117f-688b-4f6f-8597-755788182771","Type":"ContainerDied","Data":"2d6abdf1dd7f8cfaac48431629a333904385d342779c6a7c3318315ef6a825d6"}
Jul 14 12:15:55.583095 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:55.582639    2551 scope.go:117] "RemoveContainer" containerID="e77efb6a2ea32ec7689d2033ec3cfaa27be800afb7519bcabaaf5ea9b3f94cc1"
Jul 14 12:15:55.591505 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:55.591481    2551 scope.go:117] "RemoveContainer" containerID="e77efb6a2ea32ec7689d2033ec3cfaa27be800afb7519bcabaaf5ea9b3f94cc1"
Jul 14 12:15:55.591807 ip-10-0-136-173 kubenswrapper[2551]: E0714 12:15:55.591787    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e77efb6a2ea32ec7689d2033ec3cfaa27be800afb7519bcabaaf5ea9b3f94cc1\": container with ID starting with e77efb6a2ea32ec7689d2033ec3cfaa27be800afb7519bcabaaf5ea9b3f94cc1 not found: ID does not exist" containerID="e77efb6a2ea32ec7689d2033ec3cfaa27be800afb7519bcabaaf5ea9b3f94cc1"
Jul 14 12:15:55.591873 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:55.591818    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e77efb6a2ea32ec7689d2033ec3cfaa27be800afb7519bcabaaf5ea9b3f94cc1"} err="failed to get container status \"e77efb6a2ea32ec7689d2033ec3cfaa27be800afb7519bcabaaf5ea9b3f94cc1\": rpc error: code = NotFound desc = could not find container \"e77efb6a2ea32ec7689d2033ec3cfaa27be800afb7519bcabaaf5ea9b3f94cc1\": container with ID starting with e77efb6a2ea32ec7689d2033ec3cfaa27be800afb7519bcabaaf5ea9b3f94cc1 not found: ID does not exist"
Jul 14 12:15:55.602060 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:55.602030    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ae12a0-5b98b64dd4-4bxg7"]
Jul 14 12:15:55.605385 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:55.605362    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ae12a0-5b98b64dd4-4bxg7"]
Jul 14 12:15:57.191594 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:57.191550    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40cb117f-688b-4f6f-8597-755788182771" path="/var/lib/kubelet/pods/40cb117f-688b-4f6f-8597-755788182771/volumes"
Jul 14 12:15:57.963897 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:57.963853    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-ae12a0/test-model-ae12a0-kserve-745db7bdbf-2wzhl" podUID="671c4f06-dc99-4c32-a2e9-21a49361ae47" containerName="main" containerID="cri-o://f794a7e0d219d7f36a6f2fc82b2f1d01528ac791a0d06a867d5a71c8d9bc686d" gracePeriod=45
Jul 14 12:15:58.221905 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:58.221831    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ae12a0/test-model-ae12a0-kserve-745db7bdbf-2wzhl"
Jul 14 12:15:58.319411 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:58.319365    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/671c4f06-dc99-4c32-a2e9-21a49361ae47-tmp-dir\") pod \"671c4f06-dc99-4c32-a2e9-21a49361ae47\" (UID: \"671c4f06-dc99-4c32-a2e9-21a49361ae47\") "
Jul 14 12:15:58.319623 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:58.319438    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vdfvb\" (UniqueName: \"kubernetes.io/projected/671c4f06-dc99-4c32-a2e9-21a49361ae47-kube-api-access-vdfvb\") pod \"671c4f06-dc99-4c32-a2e9-21a49361ae47\" (UID: \"671c4f06-dc99-4c32-a2e9-21a49361ae47\") "
Jul 14 12:15:58.319623 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:58.319468    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/671c4f06-dc99-4c32-a2e9-21a49361ae47-dshm\") pod \"671c4f06-dc99-4c32-a2e9-21a49361ae47\" (UID: \"671c4f06-dc99-4c32-a2e9-21a49361ae47\") "
Jul 14 12:15:58.319623 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:58.319514    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/671c4f06-dc99-4c32-a2e9-21a49361ae47-model-cache\") pod \"671c4f06-dc99-4c32-a2e9-21a49361ae47\" (UID: \"671c4f06-dc99-4c32-a2e9-21a49361ae47\") "
Jul 14 12:15:58.319623 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:58.319558    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/671c4f06-dc99-4c32-a2e9-21a49361ae47-tls-certs\") pod \"671c4f06-dc99-4c32-a2e9-21a49361ae47\" (UID: \"671c4f06-dc99-4c32-a2e9-21a49361ae47\") "
Jul 14 12:15:58.319855 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:58.319624    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/671c4f06-dc99-4c32-a2e9-21a49361ae47-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "671c4f06-dc99-4c32-a2e9-21a49361ae47" (UID: "671c4f06-dc99-4c32-a2e9-21a49361ae47"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 12:15:58.319902 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:58.319823    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/671c4f06-dc99-4c32-a2e9-21a49361ae47-model-cache" (OuterVolumeSpecName: "model-cache") pod "671c4f06-dc99-4c32-a2e9-21a49361ae47" (UID: "671c4f06-dc99-4c32-a2e9-21a49361ae47"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 12:15:58.320138 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:58.319829    2551 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/671c4f06-dc99-4c32-a2e9-21a49361ae47-tmp-dir\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 12:15:58.327001 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:58.322423    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/671c4f06-dc99-4c32-a2e9-21a49361ae47-dshm" (OuterVolumeSpecName: "dshm") pod "671c4f06-dc99-4c32-a2e9-21a49361ae47" (UID: "671c4f06-dc99-4c32-a2e9-21a49361ae47"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 12:15:58.327001 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:58.322697    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/671c4f06-dc99-4c32-a2e9-21a49361ae47-kube-api-access-vdfvb" (OuterVolumeSpecName: "kube-api-access-vdfvb") pod "671c4f06-dc99-4c32-a2e9-21a49361ae47" (UID: "671c4f06-dc99-4c32-a2e9-21a49361ae47"). InnerVolumeSpecName "kube-api-access-vdfvb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 12:15:58.327770 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:58.327742    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/671c4f06-dc99-4c32-a2e9-21a49361ae47-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "671c4f06-dc99-4c32-a2e9-21a49361ae47" (UID: "671c4f06-dc99-4c32-a2e9-21a49361ae47"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 12:15:58.420520 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:58.420467    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/671c4f06-dc99-4c32-a2e9-21a49361ae47-home\") pod \"671c4f06-dc99-4c32-a2e9-21a49361ae47\" (UID: \"671c4f06-dc99-4c32-a2e9-21a49361ae47\") "
Jul 14 12:15:58.420747 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:58.420719    2551 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/671c4f06-dc99-4c32-a2e9-21a49361ae47-model-cache\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 12:15:58.420747 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:58.420735    2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/671c4f06-dc99-4c32-a2e9-21a49361ae47-tls-certs\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 12:15:58.420747 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:58.420744    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdfvb\" (UniqueName: \"kubernetes.io/projected/671c4f06-dc99-4c32-a2e9-21a49361ae47-kube-api-access-vdfvb\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 12:15:58.420855 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:58.420753    2551 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/671c4f06-dc99-4c32-a2e9-21a49361ae47-dshm\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 12:15:58.420855 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:58.420754    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/671c4f06-dc99-4c32-a2e9-21a49361ae47-home" (OuterVolumeSpecName: "home") pod "671c4f06-dc99-4c32-a2e9-21a49361ae47" (UID: "671c4f06-dc99-4c32-a2e9-21a49361ae47"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 12:15:58.521688 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:58.521545    2551 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/671c4f06-dc99-4c32-a2e9-21a49361ae47-home\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 12:15:58.599146 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:58.599104    2551 generic.go:356] "Generic (PLEG): container finished" podID="671c4f06-dc99-4c32-a2e9-21a49361ae47" containerID="f794a7e0d219d7f36a6f2fc82b2f1d01528ac791a0d06a867d5a71c8d9bc686d" exitCode=0
Jul 14 12:15:58.599350 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:58.599153    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ae12a0/test-model-ae12a0-kserve-745db7bdbf-2wzhl" event={"ID":"671c4f06-dc99-4c32-a2e9-21a49361ae47","Type":"ContainerDied","Data":"f794a7e0d219d7f36a6f2fc82b2f1d01528ac791a0d06a867d5a71c8d9bc686d"}
Jul 14 12:15:58.599350 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:58.599176    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ae12a0/test-model-ae12a0-kserve-745db7bdbf-2wzhl"
Jul 14 12:15:58.599350 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:58.599199    2551 scope.go:117] "RemoveContainer" containerID="f794a7e0d219d7f36a6f2fc82b2f1d01528ac791a0d06a867d5a71c8d9bc686d"
Jul 14 12:15:58.599350 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:58.599186    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ae12a0/test-model-ae12a0-kserve-745db7bdbf-2wzhl" event={"ID":"671c4f06-dc99-4c32-a2e9-21a49361ae47","Type":"ContainerDied","Data":"a3a210522184e61d80d87f276f48267ec9117577dcee53463e9685a986ef5e81"}
Jul 14 12:15:58.608231 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:58.608212    2551 scope.go:117] "RemoveContainer" containerID="f794a7e0d219d7f36a6f2fc82b2f1d01528ac791a0d06a867d5a71c8d9bc686d"
Jul 14 12:15:58.608546 ip-10-0-136-173 kubenswrapper[2551]: E0714 12:15:58.608525    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f794a7e0d219d7f36a6f2fc82b2f1d01528ac791a0d06a867d5a71c8d9bc686d\": container with ID starting with f794a7e0d219d7f36a6f2fc82b2f1d01528ac791a0d06a867d5a71c8d9bc686d not found: ID does not exist" containerID="f794a7e0d219d7f36a6f2fc82b2f1d01528ac791a0d06a867d5a71c8d9bc686d"
Jul 14 12:15:58.608617 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:58.608555    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f794a7e0d219d7f36a6f2fc82b2f1d01528ac791a0d06a867d5a71c8d9bc686d"} err="failed to get container status \"f794a7e0d219d7f36a6f2fc82b2f1d01528ac791a0d06a867d5a71c8d9bc686d\": rpc error: code = NotFound desc = could not find container \"f794a7e0d219d7f36a6f2fc82b2f1d01528ac791a0d06a867d5a71c8d9bc686d\": container with ID starting with f794a7e0d219d7f36a6f2fc82b2f1d01528ac791a0d06a867d5a71c8d9bc686d not found: ID does not exist"
Jul 14 12:15:58.623316 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:58.623287    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-ae12a0/test-model-ae12a0-kserve-745db7bdbf-2wzhl"]
Jul 14 12:15:58.635311 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:58.635277    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-ae12a0/test-model-ae12a0-kserve-745db7bdbf-2wzhl"]
Jul 14 12:15:58.932654 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:58.932505    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-a015ba/test-model-a015ba-kserve-5ddf59d5-8f5zb" podUID="e3e97ba9-2c8b-41c8-813f-72eb1fee24e1" containerName="main" containerID="cri-o://d8c8f44b8626c2507f5178f99c1fbeff281149c4ed8fdfd86f8d9c4e7fb54a6c" gracePeriod=45
Jul 14 12:15:59.191382 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:59.191334    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="671c4f06-dc99-4c32-a2e9-21a49361ae47" path="/var/lib/kubelet/pods/671c4f06-dc99-4c32-a2e9-21a49361ae47/volumes"
Jul 14 12:15:59.193970 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:59.193949    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a015ba/test-model-a015ba-kserve-5ddf59d5-8f5zb"
Jul 14 12:15:59.214222 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:59.214191    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:15:59.226872 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:59.226842    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3e97ba9-2c8b-41c8-813f-72eb1fee24e1-tls-certs\") pod \"e3e97ba9-2c8b-41c8-813f-72eb1fee24e1\" (UID: \"e3e97ba9-2c8b-41c8-813f-72eb1fee24e1\") "
Jul 14 12:15:59.227358 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:59.227007    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ss756\" (UniqueName: \"kubernetes.io/projected/e3e97ba9-2c8b-41c8-813f-72eb1fee24e1-kube-api-access-ss756\") pod \"e3e97ba9-2c8b-41c8-813f-72eb1fee24e1\" (UID: \"e3e97ba9-2c8b-41c8-813f-72eb1fee24e1\") "
Jul 14 12:15:59.227358 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:59.227097    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3e97ba9-2c8b-41c8-813f-72eb1fee24e1-model-cache\") pod \"e3e97ba9-2c8b-41c8-813f-72eb1fee24e1\" (UID: \"e3e97ba9-2c8b-41c8-813f-72eb1fee24e1\") "
Jul 14 12:15:59.227358 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:59.227144    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3e97ba9-2c8b-41c8-813f-72eb1fee24e1-dshm\") pod \"e3e97ba9-2c8b-41c8-813f-72eb1fee24e1\" (UID: \"e3e97ba9-2c8b-41c8-813f-72eb1fee24e1\") "
Jul 14 12:15:59.227358 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:59.227179    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3e97ba9-2c8b-41c8-813f-72eb1fee24e1-tmp-dir\") pod \"e3e97ba9-2c8b-41c8-813f-72eb1fee24e1\" (UID: \"e3e97ba9-2c8b-41c8-813f-72eb1fee24e1\") "
Jul 14 12:15:59.227358 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:59.227201    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3e97ba9-2c8b-41c8-813f-72eb1fee24e1-home\") pod \"e3e97ba9-2c8b-41c8-813f-72eb1fee24e1\" (UID: \"e3e97ba9-2c8b-41c8-813f-72eb1fee24e1\") "
Jul 14 12:15:59.230515 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:59.227751    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3e97ba9-2c8b-41c8-813f-72eb1fee24e1-home" (OuterVolumeSpecName: "home") pod "e3e97ba9-2c8b-41c8-813f-72eb1fee24e1" (UID: "e3e97ba9-2c8b-41c8-813f-72eb1fee24e1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 12:15:59.230515 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:59.228967    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3e97ba9-2c8b-41c8-813f-72eb1fee24e1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e3e97ba9-2c8b-41c8-813f-72eb1fee24e1" (UID: "e3e97ba9-2c8b-41c8-813f-72eb1fee24e1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 12:15:59.230515 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:59.229037    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3e97ba9-2c8b-41c8-813f-72eb1fee24e1-model-cache" (OuterVolumeSpecName: "model-cache") pod "e3e97ba9-2c8b-41c8-813f-72eb1fee24e1" (UID: "e3e97ba9-2c8b-41c8-813f-72eb1fee24e1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 12:15:59.231388 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:59.231351    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3e97ba9-2c8b-41c8-813f-72eb1fee24e1-kube-api-access-ss756" (OuterVolumeSpecName: "kube-api-access-ss756") pod "e3e97ba9-2c8b-41c8-813f-72eb1fee24e1" (UID: "e3e97ba9-2c8b-41c8-813f-72eb1fee24e1"). InnerVolumeSpecName "kube-api-access-ss756". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 12:15:59.231504 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:59.231435    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3e97ba9-2c8b-41c8-813f-72eb1fee24e1-dshm" (OuterVolumeSpecName: "dshm") pod "e3e97ba9-2c8b-41c8-813f-72eb1fee24e1" (UID: "e3e97ba9-2c8b-41c8-813f-72eb1fee24e1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 12:15:59.231605 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:59.231500    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3e97ba9-2c8b-41c8-813f-72eb1fee24e1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e3e97ba9-2c8b-41c8-813f-72eb1fee24e1" (UID: "e3e97ba9-2c8b-41c8-813f-72eb1fee24e1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 12:15:59.328011 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:59.327970    2551 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3e97ba9-2c8b-41c8-813f-72eb1fee24e1-dshm\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 12:15:59.328011 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:59.328004    2551 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3e97ba9-2c8b-41c8-813f-72eb1fee24e1-tmp-dir\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 12:15:59.328011 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:59.328012    2551 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3e97ba9-2c8b-41c8-813f-72eb1fee24e1-home\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 12:15:59.328011 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:59.328020    2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3e97ba9-2c8b-41c8-813f-72eb1fee24e1-tls-certs\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 12:15:59.328382 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:59.328029    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ss756\" (UniqueName: \"kubernetes.io/projected/e3e97ba9-2c8b-41c8-813f-72eb1fee24e1-kube-api-access-ss756\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 12:15:59.328382 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:59.328040    2551 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3e97ba9-2c8b-41c8-813f-72eb1fee24e1-model-cache\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 12:15:59.605012 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:59.604916    2551 generic.go:356] "Generic (PLEG): container finished" podID="e3e97ba9-2c8b-41c8-813f-72eb1fee24e1" containerID="d8c8f44b8626c2507f5178f99c1fbeff281149c4ed8fdfd86f8d9c4e7fb54a6c" exitCode=0
Jul 14 12:15:59.605012 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:59.604959    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a015ba/test-model-a015ba-kserve-5ddf59d5-8f5zb"
Jul 14 12:15:59.605260 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:59.605001    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a015ba/test-model-a015ba-kserve-5ddf59d5-8f5zb" event={"ID":"e3e97ba9-2c8b-41c8-813f-72eb1fee24e1","Type":"ContainerDied","Data":"d8c8f44b8626c2507f5178f99c1fbeff281149c4ed8fdfd86f8d9c4e7fb54a6c"}
Jul 14 12:15:59.605260 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:59.605051    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a015ba/test-model-a015ba-kserve-5ddf59d5-8f5zb" event={"ID":"e3e97ba9-2c8b-41c8-813f-72eb1fee24e1","Type":"ContainerDied","Data":"62eea4cf6c00236edf5c4dd9d36f684906531bc8e451b506e912328ef449e8e7"}
Jul 14 12:15:59.605260 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:59.605072    2551 scope.go:117] "RemoveContainer" containerID="d8c8f44b8626c2507f5178f99c1fbeff281149c4ed8fdfd86f8d9c4e7fb54a6c"
Jul 14 12:15:59.614360 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:59.614335    2551 scope.go:117] "RemoveContainer" containerID="d8c8f44b8626c2507f5178f99c1fbeff281149c4ed8fdfd86f8d9c4e7fb54a6c"
Jul 14 12:15:59.614672 ip-10-0-136-173 kubenswrapper[2551]: E0714 12:15:59.614649    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8c8f44b8626c2507f5178f99c1fbeff281149c4ed8fdfd86f8d9c4e7fb54a6c\": container with ID starting with d8c8f44b8626c2507f5178f99c1fbeff281149c4ed8fdfd86f8d9c4e7fb54a6c not found: ID does not exist" containerID="d8c8f44b8626c2507f5178f99c1fbeff281149c4ed8fdfd86f8d9c4e7fb54a6c"
Jul 14 12:15:59.614744 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:59.614680    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8c8f44b8626c2507f5178f99c1fbeff281149c4ed8fdfd86f8d9c4e7fb54a6c"} err="failed to get container status \"d8c8f44b8626c2507f5178f99c1fbeff281149c4ed8fdfd86f8d9c4e7fb54a6c\": rpc error: code = NotFound desc = could not find container \"d8c8f44b8626c2507f5178f99c1fbeff281149c4ed8fdfd86f8d9c4e7fb54a6c\": container with ID starting with d8c8f44b8626c2507f5178f99c1fbeff281149c4ed8fdfd86f8d9c4e7fb54a6c not found: ID does not exist"
Jul 14 12:15:59.631955 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:59.631919    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-a015ba/test-model-a015ba-kserve-5ddf59d5-8f5zb"]
Jul 14 12:15:59.639115 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:15:59.639086    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-a015ba/test-model-a015ba-kserve-5ddf59d5-8f5zb"]
Jul 14 12:16:01.192511 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:01.192473    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3e97ba9-2c8b-41c8-813f-72eb1fee24e1" path="/var/lib/kubelet/pods/e3e97ba9-2c8b-41c8-813f-72eb1fee24e1/volumes"
Jul 14 12:16:04.690950 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:04.690909    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 12:16:06.752968 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:06.752931    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:16:06.971044 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:06.971007    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:16:08.506664 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:08.506623    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:16:08.607452 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:08.607418    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:16:09.917663 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:09.917632    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:16:10.311828 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:10.311789    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:16:10.958635 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:10.958595    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:16:11.060771 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:11.060740    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:16:20.784470 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:20.784362    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a015ba-69c489d9f7-ffwkz"]
Jul 14 12:16:20.784957 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:20.784654    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a015ba-69c489d9f7-ffwkz" podUID="eae5d888-02f4-462c-99f4-3c9a805ff627" containerName="maas-api" containerID="cri-o://ad92042d522160e73a2aa501c672b4147bf8af663c2d729a30cd9512615e5038" gracePeriod=30
Jul 14 12:16:20.837143 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:20.837112    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:16:21.043950 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:21.043885    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a015ba-69c489d9f7-ffwkz"
Jul 14 12:16:21.111713 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:21.111673    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eae5d888-02f4-462c-99f4-3c9a805ff627-maas-api-tls\") pod \"eae5d888-02f4-462c-99f4-3c9a805ff627\" (UID: \"eae5d888-02f4-462c-99f4-3c9a805ff627\") "
Jul 14 12:16:21.111921 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:21.111743    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7srd2\" (UniqueName: \"kubernetes.io/projected/eae5d888-02f4-462c-99f4-3c9a805ff627-kube-api-access-7srd2\") pod \"eae5d888-02f4-462c-99f4-3c9a805ff627\" (UID: \"eae5d888-02f4-462c-99f4-3c9a805ff627\") "
Jul 14 12:16:21.111921 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:21.111773    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eae5d888-02f4-462c-99f4-3c9a805ff627-openshift-service-ca\") pod \"eae5d888-02f4-462c-99f4-3c9a805ff627\" (UID: \"eae5d888-02f4-462c-99f4-3c9a805ff627\") "
Jul 14 12:16:21.112220 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:21.112192    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eae5d888-02f4-462c-99f4-3c9a805ff627-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "eae5d888-02f4-462c-99f4-3c9a805ff627" (UID: "eae5d888-02f4-462c-99f4-3c9a805ff627"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 12:16:21.113950 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:21.113922    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eae5d888-02f4-462c-99f4-3c9a805ff627-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "eae5d888-02f4-462c-99f4-3c9a805ff627" (UID: "eae5d888-02f4-462c-99f4-3c9a805ff627"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 12:16:21.114039 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:21.113973    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eae5d888-02f4-462c-99f4-3c9a805ff627-kube-api-access-7srd2" (OuterVolumeSpecName: "kube-api-access-7srd2") pod "eae5d888-02f4-462c-99f4-3c9a805ff627" (UID: "eae5d888-02f4-462c-99f4-3c9a805ff627"). InnerVolumeSpecName "kube-api-access-7srd2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 12:16:21.213325 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:21.213288    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7srd2\" (UniqueName: \"kubernetes.io/projected/eae5d888-02f4-462c-99f4-3c9a805ff627-kube-api-access-7srd2\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 12:16:21.213325 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:21.213324    2551 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eae5d888-02f4-462c-99f4-3c9a805ff627-openshift-service-ca\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 12:16:21.213543 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:21.213338    2551 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eae5d888-02f4-462c-99f4-3c9a805ff627-maas-api-tls\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 12:16:21.716208 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:21.716167    2551 generic.go:356] "Generic (PLEG): container finished" podID="eae5d888-02f4-462c-99f4-3c9a805ff627" containerID="ad92042d522160e73a2aa501c672b4147bf8af663c2d729a30cd9512615e5038" exitCode=0
Jul 14 12:16:21.716208 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:21.716214    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a015ba-69c489d9f7-ffwkz"
Jul 14 12:16:21.716455 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:21.716252    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a015ba-69c489d9f7-ffwkz" event={"ID":"eae5d888-02f4-462c-99f4-3c9a805ff627","Type":"ContainerDied","Data":"ad92042d522160e73a2aa501c672b4147bf8af663c2d729a30cd9512615e5038"}
Jul 14 12:16:21.716455 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:21.716309    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a015ba-69c489d9f7-ffwkz" event={"ID":"eae5d888-02f4-462c-99f4-3c9a805ff627","Type":"ContainerDied","Data":"5f18d5c41dd66490ad8f8c744830633e6fb89ae10c30bf61fbf40c2cf252325e"}
Jul 14 12:16:21.716455 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:21.716333    2551 scope.go:117] "RemoveContainer" containerID="ad92042d522160e73a2aa501c672b4147bf8af663c2d729a30cd9512615e5038"
Jul 14 12:16:21.725091 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:21.725069    2551 scope.go:117] "RemoveContainer" containerID="ad92042d522160e73a2aa501c672b4147bf8af663c2d729a30cd9512615e5038"
Jul 14 12:16:21.725429 ip-10-0-136-173 kubenswrapper[2551]: E0714 12:16:21.725400    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad92042d522160e73a2aa501c672b4147bf8af663c2d729a30cd9512615e5038\": container with ID starting with ad92042d522160e73a2aa501c672b4147bf8af663c2d729a30cd9512615e5038 not found: ID does not exist" containerID="ad92042d522160e73a2aa501c672b4147bf8af663c2d729a30cd9512615e5038"
Jul 14 12:16:21.725529 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:21.725438    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad92042d522160e73a2aa501c672b4147bf8af663c2d729a30cd9512615e5038"} err="failed to get container status \"ad92042d522160e73a2aa501c672b4147bf8af663c2d729a30cd9512615e5038\": rpc error: code = NotFound desc = could not find container \"ad92042d522160e73a2aa501c672b4147bf8af663c2d729a30cd9512615e5038\": container with ID starting with ad92042d522160e73a2aa501c672b4147bf8af663c2d729a30cd9512615e5038 not found: ID does not exist"
Jul 14 12:16:21.736190 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:21.736139    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a015ba-69c489d9f7-ffwkz"]
Jul 14 12:16:21.740945 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:21.740908    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a015ba-69c489d9f7-ffwkz"]
Jul 14 12:16:23.191489 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:23.191453    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eae5d888-02f4-462c-99f4-3c9a805ff627" path="/var/lib/kubelet/pods/eae5d888-02f4-462c-99f4-3c9a805ff627/volumes"
Jul 14 12:16:24.001818 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:23.998040    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 12:16:25.399404 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:25.399368    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:16:28.243474 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:28.243371    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:16:28.342205 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:28.342173    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:16:32.693940 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:32.693900    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:16:34.144983 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:34.144948    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:16:36.140396 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:36.140361    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:16:36.358632 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:36.358573    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:16:36.719796 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:36.719762    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:16:37.220435 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:37.220402    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:16:37.387227 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:37.387192    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:16:39.691880 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:39.691845    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:16:39.746700 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:39.746662    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3ce01d-57f9d5b6f9-9pvz5"]
Jul 14 12:16:39.747017 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:39.747003    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eae5d888-02f4-462c-99f4-3c9a805ff627" containerName="maas-api"
Jul 14 12:16:39.747077 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:39.747019    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="eae5d888-02f4-462c-99f4-3c9a805ff627" containerName="maas-api"
Jul 14 12:16:39.747077 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:39.747033    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3e97ba9-2c8b-41c8-813f-72eb1fee24e1" containerName="main"
Jul 14 12:16:39.747077 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:39.747039    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3e97ba9-2c8b-41c8-813f-72eb1fee24e1" containerName="main"
Jul 14 12:16:39.747077 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:39.747047    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="671c4f06-dc99-4c32-a2e9-21a49361ae47" containerName="main"
Jul 14 12:16:39.747077 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:39.747052    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="671c4f06-dc99-4c32-a2e9-21a49361ae47" containerName="main"
Jul 14 12:16:39.747077 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:39.747068    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40cb117f-688b-4f6f-8597-755788182771" containerName="maas-api"
Jul 14 12:16:39.747077 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:39.747074    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="40cb117f-688b-4f6f-8597-755788182771" containerName="maas-api"
Jul 14 12:16:39.747308 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:39.747130    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3e97ba9-2c8b-41c8-813f-72eb1fee24e1" containerName="main"
Jul 14 12:16:39.747308 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:39.747140    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="40cb117f-688b-4f6f-8597-755788182771" containerName="maas-api"
Jul 14 12:16:39.747308 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:39.747146    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="eae5d888-02f4-462c-99f4-3c9a805ff627" containerName="maas-api"
Jul 14 12:16:39.747308 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:39.747152    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="671c4f06-dc99-4c32-a2e9-21a49361ae47" containerName="main"
Jul 14 12:16:39.750123 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:39.750101    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3ce01d-57f9d5b6f9-9pvz5"
Jul 14 12:16:39.753219 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:39.753191    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-62k4x\""
Jul 14 12:16:39.753219 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:39.753208    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 14 12:16:39.753433 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:39.753213    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 14 12:16:39.754342 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:39.754316    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-3ce01d-serving-cert\""
Jul 14 12:16:39.763559 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:39.763535    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3ce01d-57f9d5b6f9-9pvz5"]
Jul 14 12:16:39.796713 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:39.796671    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:16:39.877671 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:39.877629    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81507b85-3f90-462b-9ec2-07e6785b8c9a-maas-api-tls\") pod \"maas-api-e2e-api-a-3ce01d-57f9d5b6f9-9pvz5\" (UID: \"81507b85-3f90-462b-9ec2-07e6785b8c9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3ce01d-57f9d5b6f9-9pvz5"
Jul 14 12:16:39.877671 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:39.877666    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zk62n\" (UniqueName: \"kubernetes.io/projected/81507b85-3f90-462b-9ec2-07e6785b8c9a-kube-api-access-zk62n\") pod \"maas-api-e2e-api-a-3ce01d-57f9d5b6f9-9pvz5\" (UID: \"81507b85-3f90-462b-9ec2-07e6785b8c9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3ce01d-57f9d5b6f9-9pvz5"
Jul 14 12:16:39.877962 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:39.877713    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/81507b85-3f90-462b-9ec2-07e6785b8c9a-openshift-service-ca\") pod \"maas-api-e2e-api-a-3ce01d-57f9d5b6f9-9pvz5\" (UID: \"81507b85-3f90-462b-9ec2-07e6785b8c9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3ce01d-57f9d5b6f9-9pvz5"
Jul 14 12:16:39.915555 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:39.915519    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:16:39.978397 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:39.978302    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/81507b85-3f90-462b-9ec2-07e6785b8c9a-openshift-service-ca\") pod \"maas-api-e2e-api-a-3ce01d-57f9d5b6f9-9pvz5\" (UID: \"81507b85-3f90-462b-9ec2-07e6785b8c9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3ce01d-57f9d5b6f9-9pvz5"
Jul 14 12:16:39.978554 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:39.978468    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81507b85-3f90-462b-9ec2-07e6785b8c9a-maas-api-tls\") pod \"maas-api-e2e-api-a-3ce01d-57f9d5b6f9-9pvz5\" (UID: \"81507b85-3f90-462b-9ec2-07e6785b8c9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3ce01d-57f9d5b6f9-9pvz5"
Jul 14 12:16:39.978554 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:39.978506    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk62n\" (UniqueName: \"kubernetes.io/projected/81507b85-3f90-462b-9ec2-07e6785b8c9a-kube-api-access-zk62n\") pod \"maas-api-e2e-api-a-3ce01d-57f9d5b6f9-9pvz5\" (UID: \"81507b85-3f90-462b-9ec2-07e6785b8c9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3ce01d-57f9d5b6f9-9pvz5"
Jul 14 12:16:39.979157 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:39.979138    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/81507b85-3f90-462b-9ec2-07e6785b8c9a-openshift-service-ca\") pod \"maas-api-e2e-api-a-3ce01d-57f9d5b6f9-9pvz5\" (UID: \"81507b85-3f90-462b-9ec2-07e6785b8c9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3ce01d-57f9d5b6f9-9pvz5"
Jul 14 12:16:39.981069 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:39.981052    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81507b85-3f90-462b-9ec2-07e6785b8c9a-maas-api-tls\") pod \"maas-api-e2e-api-a-3ce01d-57f9d5b6f9-9pvz5\" (UID: \"81507b85-3f90-462b-9ec2-07e6785b8c9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3ce01d-57f9d5b6f9-9pvz5"
Jul 14 12:16:39.989315 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:39.989284    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk62n\" (UniqueName: \"kubernetes.io/projected/81507b85-3f90-462b-9ec2-07e6785b8c9a-kube-api-access-zk62n\") pod \"maas-api-e2e-api-a-3ce01d-57f9d5b6f9-9pvz5\" (UID: \"81507b85-3f90-462b-9ec2-07e6785b8c9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3ce01d-57f9d5b6f9-9pvz5"
Jul 14 12:16:40.060590 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:40.060552    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:16:40.061398 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:40.061384    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3ce01d-57f9d5b6f9-9pvz5"
Jul 14 12:16:40.216687 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:40.216663    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3ce01d-57f9d5b6f9-9pvz5"]
Jul 14 12:16:40.219666 ip-10-0-136-173 kubenswrapper[2551]: W0714 12:16:40.219637    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81507b85_3f90_462b_9ec2_07e6785b8c9a.slice/crio-dcd43e3ca51f93391b86a6ebd99068f59e1d3f43ab3d5fab0c9810f6be4eb23b WatchSource:0}: Error finding container dcd43e3ca51f93391b86a6ebd99068f59e1d3f43ab3d5fab0c9810f6be4eb23b: Status 404 returned error can't find the container with id dcd43e3ca51f93391b86a6ebd99068f59e1d3f43ab3d5fab0c9810f6be4eb23b
Jul 14 12:16:40.221444 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:40.221425    2551 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 14 12:16:40.518443 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:40.518406    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:16:40.689538 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:40.689484    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:16:40.804824 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:40.804735    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3ce01d-57f9d5b6f9-9pvz5" event={"ID":"81507b85-3f90-462b-9ec2-07e6785b8c9a","Type":"ContainerStarted","Data":"dcd43e3ca51f93391b86a6ebd99068f59e1d3f43ab3d5fab0c9810f6be4eb23b"}
Jul 14 12:16:41.810559 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:41.810519    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3ce01d-57f9d5b6f9-9pvz5" event={"ID":"81507b85-3f90-462b-9ec2-07e6785b8c9a","Type":"ContainerStarted","Data":"e0bd1e0e039f00ca981bc9733eff32c158a67d9634b9abbf9967326653d936b1"}
Jul 14 12:16:41.811093 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:41.810660    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3ce01d-57f9d5b6f9-9pvz5"
Jul 14 12:16:41.831195 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:41.831142    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3ce01d-57f9d5b6f9-9pvz5" podStartSLOduration=2.011201117 podStartE2EDuration="2.831126084s" totalImagesPullingTime="819.924967ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 12:16:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 12:16:41.829465167 +0000 UTC m=+2353.061274490" watchObservedRunningTime="2026-07-14 12:16:41.831126084 +0000 UTC m=+2353.062935380"
Jul 14 12:16:44.104225 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:44.104185    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 12:16:44.212437 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:44.212396    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:16:44.719769 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:44.719731    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:16:44.980204 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:44.980119    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-808361-openshift-default-7cf7d4f7fc-4j6l9"]
Jul 14 12:16:44.983695 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:44.983672    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-808361-openshift-default-7cf7d4f7fc-4j6l9"
Jul 14 12:16:44.986358 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:44.986333    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-808361-openshift-default-dockercfg-ndmkm\""
Jul 14 12:16:44.997510 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:44.997478    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-808361-openshift-default-7cf7d4f7fc-4j6l9"]
Jul 14 12:16:45.051020 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:45.050985    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:16:45.123979 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:45.123933    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6a34c02-28ba-4bde-8382-1b7d301285da-istio-data\") pod \"e2e-api-b-808361-openshift-default-7cf7d4f7fc-4j6l9\" (UID: \"b6a34c02-28ba-4bde-8382-1b7d301285da\") " pod="openshift-ingress/e2e-api-b-808361-openshift-default-7cf7d4f7fc-4j6l9"
Jul 14 12:16:45.123979 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:45.123972    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6a34c02-28ba-4bde-8382-1b7d301285da-istio-token\") pod \"e2e-api-b-808361-openshift-default-7cf7d4f7fc-4j6l9\" (UID: \"b6a34c02-28ba-4bde-8382-1b7d301285da\") " pod="openshift-ingress/e2e-api-b-808361-openshift-default-7cf7d4f7fc-4j6l9"
Jul 14 12:16:45.124559 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:45.124003    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6a34c02-28ba-4bde-8382-1b7d301285da-istiod-ca-cert\") pod \"e2e-api-b-808361-openshift-default-7cf7d4f7fc-4j6l9\" (UID: \"b6a34c02-28ba-4bde-8382-1b7d301285da\") " pod="openshift-ingress/e2e-api-b-808361-openshift-default-7cf7d4f7fc-4j6l9"
Jul 14 12:16:45.124559 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:45.124138    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6a34c02-28ba-4bde-8382-1b7d301285da-credential-socket\") pod \"e2e-api-b-808361-openshift-default-7cf7d4f7fc-4j6l9\" (UID: \"b6a34c02-28ba-4bde-8382-1b7d301285da\") " pod="openshift-ingress/e2e-api-b-808361-openshift-default-7cf7d4f7fc-4j6l9"
Jul 14 12:16:45.124559 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:45.124211    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6a34c02-28ba-4bde-8382-1b7d301285da-workload-socket\") pod \"e2e-api-b-808361-openshift-default-7cf7d4f7fc-4j6l9\" (UID: \"b6a34c02-28ba-4bde-8382-1b7d301285da\") " pod="openshift-ingress/e2e-api-b-808361-openshift-default-7cf7d4f7fc-4j6l9"
Jul 14 12:16:45.124559 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:45.124240    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6a34c02-28ba-4bde-8382-1b7d301285da-istio-podinfo\") pod \"e2e-api-b-808361-openshift-default-7cf7d4f7fc-4j6l9\" (UID: \"b6a34c02-28ba-4bde-8382-1b7d301285da\") " pod="openshift-ingress/e2e-api-b-808361-openshift-default-7cf7d4f7fc-4j6l9"
Jul 14 12:16:45.124559 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:45.124270    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6a34c02-28ba-4bde-8382-1b7d301285da-istio-envoy\") pod \"e2e-api-b-808361-openshift-default-7cf7d4f7fc-4j6l9\" (UID: \"b6a34c02-28ba-4bde-8382-1b7d301285da\") " pod="openshift-ingress/e2e-api-b-808361-openshift-default-7cf7d4f7fc-4j6l9"
Jul 14 12:16:45.124559 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:45.124349    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6a34c02-28ba-4bde-8382-1b7d301285da-workload-certs\") pod \"e2e-api-b-808361-openshift-default-7cf7d4f7fc-4j6l9\" (UID: \"b6a34c02-28ba-4bde-8382-1b7d301285da\") " pod="openshift-ingress/e2e-api-b-808361-openshift-default-7cf7d4f7fc-4j6l9"
Jul 14 12:16:45.124559 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:45.124381    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zfmp\" (UniqueName: \"kubernetes.io/projected/b6a34c02-28ba-4bde-8382-1b7d301285da-kube-api-access-5zfmp\") pod \"e2e-api-b-808361-openshift-default-7cf7d4f7fc-4j6l9\" (UID: \"b6a34c02-28ba-4bde-8382-1b7d301285da\") " pod="openshift-ingress/e2e-api-b-808361-openshift-default-7cf7d4f7fc-4j6l9"
Jul 14 12:16:45.225270 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:45.225219    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6a34c02-28ba-4bde-8382-1b7d301285da-workload-certs\") pod \"e2e-api-b-808361-openshift-default-7cf7d4f7fc-4j6l9\" (UID: \"b6a34c02-28ba-4bde-8382-1b7d301285da\") " pod="openshift-ingress/e2e-api-b-808361-openshift-default-7cf7d4f7fc-4j6l9"
Jul 14 12:16:45.225485 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:45.225274    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zfmp\" (UniqueName: \"kubernetes.io/projected/b6a34c02-28ba-4bde-8382-1b7d301285da-kube-api-access-5zfmp\") pod \"e2e-api-b-808361-openshift-default-7cf7d4f7fc-4j6l9\" (UID: \"b6a34c02-28ba-4bde-8382-1b7d301285da\") " pod="openshift-ingress/e2e-api-b-808361-openshift-default-7cf7d4f7fc-4j6l9"
Jul 14 12:16:45.225485 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:45.225320    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6a34c02-28ba-4bde-8382-1b7d301285da-istio-data\") pod \"e2e-api-b-808361-openshift-default-7cf7d4f7fc-4j6l9\" (UID: \"b6a34c02-28ba-4bde-8382-1b7d301285da\") " pod="openshift-ingress/e2e-api-b-808361-openshift-default-7cf7d4f7fc-4j6l9"
Jul 14 12:16:45.225485 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:45.225344    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6a34c02-28ba-4bde-8382-1b7d301285da-istio-token\") pod \"e2e-api-b-808361-openshift-default-7cf7d4f7fc-4j6l9\" (UID: \"b6a34c02-28ba-4bde-8382-1b7d301285da\") " pod="openshift-ingress/e2e-api-b-808361-openshift-default-7cf7d4f7fc-4j6l9"
Jul 14 12:16:45.225785 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:45.225748    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6a34c02-28ba-4bde-8382-1b7d301285da-workload-certs\") pod \"e2e-api-b-808361-openshift-default-7cf7d4f7fc-4j6l9\" (UID: \"b6a34c02-28ba-4bde-8382-1b7d301285da\") " pod="openshift-ingress/e2e-api-b-808361-openshift-default-7cf7d4f7fc-4j6l9"
Jul 14 12:16:45.225924 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:45.225371    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6a34c02-28ba-4bde-8382-1b7d301285da-istiod-ca-cert\") pod \"e2e-api-b-808361-openshift-default-7cf7d4f7fc-4j6l9\" (UID: \"b6a34c02-28ba-4bde-8382-1b7d301285da\") " pod="openshift-ingress/e2e-api-b-808361-openshift-default-7cf7d4f7fc-4j6l9"
Jul 14 12:16:45.225924 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:45.225896    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6a34c02-28ba-4bde-8382-1b7d301285da-istio-data\") pod \"e2e-api-b-808361-openshift-default-7cf7d4f7fc-4j6l9\" (UID: \"b6a34c02-28ba-4bde-8382-1b7d301285da\") " pod="openshift-ingress/e2e-api-b-808361-openshift-default-7cf7d4f7fc-4j6l9"
Jul 14 12:16:45.226047 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:45.225988    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6a34c02-28ba-4bde-8382-1b7d301285da-credential-socket\") pod \"e2e-api-b-808361-openshift-default-7cf7d4f7fc-4j6l9\" (UID: \"b6a34c02-28ba-4bde-8382-1b7d301285da\") " pod="openshift-ingress/e2e-api-b-808361-openshift-default-7cf7d4f7fc-4j6l9"
Jul 14 12:16:45.226108 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:45.226045    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6a34c02-28ba-4bde-8382-1b7d301285da-workload-socket\") pod \"e2e-api-b-808361-openshift-default-7cf7d4f7fc-4j6l9\" (UID: \"b6a34c02-28ba-4bde-8382-1b7d301285da\") " pod="openshift-ingress/e2e-api-b-808361-openshift-default-7cf7d4f7fc-4j6l9"
Jul 14 12:16:45.226108 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:45.226073    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6a34c02-28ba-4bde-8382-1b7d301285da-istio-podinfo\") pod \"e2e-api-b-808361-openshift-default-7cf7d4f7fc-4j6l9\" (UID: \"b6a34c02-28ba-4bde-8382-1b7d301285da\") " pod="openshift-ingress/e2e-api-b-808361-openshift-default-7cf7d4f7fc-4j6l9"
Jul 14 12:16:45.226108 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:45.226103    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6a34c02-28ba-4bde-8382-1b7d301285da-istio-envoy\") pod \"e2e-api-b-808361-openshift-default-7cf7d4f7fc-4j6l9\" (UID: \"b6a34c02-28ba-4bde-8382-1b7d301285da\") " pod="openshift-ingress/e2e-api-b-808361-openshift-default-7cf7d4f7fc-4j6l9"
Jul 14 12:16:45.226530 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:45.226484    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6a34c02-28ba-4bde-8382-1b7d301285da-istiod-ca-cert\") pod \"e2e-api-b-808361-openshift-default-7cf7d4f7fc-4j6l9\" (UID: \"b6a34c02-28ba-4bde-8382-1b7d301285da\") " pod="openshift-ingress/e2e-api-b-808361-openshift-default-7cf7d4f7fc-4j6l9"
Jul 14 12:16:45.226651 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:45.226544    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6a34c02-28ba-4bde-8382-1b7d301285da-workload-socket\") pod \"e2e-api-b-808361-openshift-default-7cf7d4f7fc-4j6l9\" (UID: \"b6a34c02-28ba-4bde-8382-1b7d301285da\") " pod="openshift-ingress/e2e-api-b-808361-openshift-default-7cf7d4f7fc-4j6l9"
Jul 14 12:16:45.226724 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:45.226686    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6a34c02-28ba-4bde-8382-1b7d301285da-credential-socket\") pod \"e2e-api-b-808361-openshift-default-7cf7d4f7fc-4j6l9\" (UID: \"b6a34c02-28ba-4bde-8382-1b7d301285da\") " pod="openshift-ingress/e2e-api-b-808361-openshift-default-7cf7d4f7fc-4j6l9"
Jul 14 12:16:45.228778 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:45.228755    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6a34c02-28ba-4bde-8382-1b7d301285da-istio-envoy\") pod \"e2e-api-b-808361-openshift-default-7cf7d4f7fc-4j6l9\" (UID: \"b6a34c02-28ba-4bde-8382-1b7d301285da\") " pod="openshift-ingress/e2e-api-b-808361-openshift-default-7cf7d4f7fc-4j6l9"
Jul 14 12:16:45.229027 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:45.229006    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6a34c02-28ba-4bde-8382-1b7d301285da-istio-podinfo\") pod \"e2e-api-b-808361-openshift-default-7cf7d4f7fc-4j6l9\" (UID: \"b6a34c02-28ba-4bde-8382-1b7d301285da\") " pod="openshift-ingress/e2e-api-b-808361-openshift-default-7cf7d4f7fc-4j6l9"
Jul 14 12:16:45.235622 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:45.235539    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6a34c02-28ba-4bde-8382-1b7d301285da-istio-token\") pod \"e2e-api-b-808361-openshift-default-7cf7d4f7fc-4j6l9\" (UID: \"b6a34c02-28ba-4bde-8382-1b7d301285da\") " pod="openshift-ingress/e2e-api-b-808361-openshift-default-7cf7d4f7fc-4j6l9"
Jul 14 12:16:45.235770 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:45.235751    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zfmp\" (UniqueName: \"kubernetes.io/projected/b6a34c02-28ba-4bde-8382-1b7d301285da-kube-api-access-5zfmp\") pod \"e2e-api-b-808361-openshift-default-7cf7d4f7fc-4j6l9\" (UID: \"b6a34c02-28ba-4bde-8382-1b7d301285da\") " pod="openshift-ingress/e2e-api-b-808361-openshift-default-7cf7d4f7fc-4j6l9"
Jul 14 12:16:45.294793 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:45.294747    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-808361-openshift-default-7cf7d4f7fc-4j6l9"
Jul 14 12:16:45.448536 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:45.448496    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-808361-openshift-default-7cf7d4f7fc-4j6l9"]
Jul 14 12:16:45.451797 ip-10-0-136-173 kubenswrapper[2551]: W0714 12:16:45.451769    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6a34c02_28ba_4bde_8382_1b7d301285da.slice/crio-f7d07da03e3577da17c340f2491cf743e14ba64c6dc33f11978f6bc6c57fda43 WatchSource:0}: Error finding container f7d07da03e3577da17c340f2491cf743e14ba64c6dc33f11978f6bc6c57fda43: Status 404 returned error can't find the container with id f7d07da03e3577da17c340f2491cf743e14ba64c6dc33f11978f6bc6c57fda43
Jul 14 12:16:45.454096 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:45.454059    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 14 12:16:45.454186 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:45.454146    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 14 12:16:45.454254 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:45.454191    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 14 12:16:45.467147 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:45.467114    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:16:45.832001 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:45.831893    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-808361-openshift-default-7cf7d4f7fc-4j6l9" event={"ID":"b6a34c02-28ba-4bde-8382-1b7d301285da","Type":"ContainerStarted","Data":"66239962ece95652c4fa8e95fb95e9285a4dfd6dede2b6ec26c1ca7c011d752c"}
Jul 14 12:16:45.832001 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:45.831956    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-808361-openshift-default-7cf7d4f7fc-4j6l9" event={"ID":"b6a34c02-28ba-4bde-8382-1b7d301285da","Type":"ContainerStarted","Data":"f7d07da03e3577da17c340f2491cf743e14ba64c6dc33f11978f6bc6c57fda43"}
Jul 14 12:16:45.858374 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:45.858323    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-808361-openshift-default-7cf7d4f7fc-4j6l9" podStartSLOduration=1.85830683 podStartE2EDuration="1.85830683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 12:16:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 12:16:45.858010167 +0000 UTC m=+2357.089819464" watchObservedRunningTime="2026-07-14 12:16:45.85830683 +0000 UTC m=+2357.090116126"
Jul 14 12:16:46.295385 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:46.295346    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-808361-openshift-default-7cf7d4f7fc-4j6l9"
Jul 14 12:16:46.300446 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:46.300414    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-808361-openshift-default-7cf7d4f7fc-4j6l9"
Jul 14 12:16:46.836819 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:46.836790    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-808361-openshift-default-7cf7d4f7fc-4j6l9"
Jul 14 12:16:46.837725 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:46.837701    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-808361-openshift-default-7cf7d4f7fc-4j6l9"
Jul 14 12:16:47.821855 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:47.821824    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3ce01d-57f9d5b6f9-9pvz5"
Jul 14 12:16:47.905337 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:47.905298    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:16:48.026815 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:48.026759    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-808361-5d568d696b-xsjm9"]
Jul 14 12:16:48.031023 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:48.030994    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-808361-5d568d696b-xsjm9"
Jul 14 12:16:48.033654 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:48.033627    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-808361-serving-cert\""
Jul 14 12:16:48.045399 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:48.045367    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-808361-5d568d696b-xsjm9"]
Jul 14 12:16:48.116392 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:48.116289    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:16:48.154398 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:48.154359    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d29605a-ac0f-4068-b11b-fec653825bee-maas-api-tls\") pod \"maas-api-e2e-api-b-808361-5d568d696b-xsjm9\" (UID: \"9d29605a-ac0f-4068-b11b-fec653825bee\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-808361-5d568d696b-xsjm9"
Jul 14 12:16:48.154624 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:48.154424    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d29605a-ac0f-4068-b11b-fec653825bee-openshift-service-ca\") pod \"maas-api-e2e-api-b-808361-5d568d696b-xsjm9\" (UID: \"9d29605a-ac0f-4068-b11b-fec653825bee\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-808361-5d568d696b-xsjm9"
Jul 14 12:16:48.154624 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:48.154473    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jtgg\" (UniqueName: \"kubernetes.io/projected/9d29605a-ac0f-4068-b11b-fec653825bee-kube-api-access-6jtgg\") pod \"maas-api-e2e-api-b-808361-5d568d696b-xsjm9\" (UID: \"9d29605a-ac0f-4068-b11b-fec653825bee\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-808361-5d568d696b-xsjm9"
Jul 14 12:16:48.255721 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:48.255676    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d29605a-ac0f-4068-b11b-fec653825bee-maas-api-tls\") pod \"maas-api-e2e-api-b-808361-5d568d696b-xsjm9\" (UID: \"9d29605a-ac0f-4068-b11b-fec653825bee\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-808361-5d568d696b-xsjm9"
Jul 14 12:16:48.255929 ip-10-0-136-173 kubenswrapper[2551]: E0714 12:16:48.255849    2551 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-808361-serving-cert: secret "maas-api-e2e-api-b-808361-serving-cert" not found
Jul 14 12:16:48.255929 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:48.255854    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d29605a-ac0f-4068-b11b-fec653825bee-openshift-service-ca\") pod \"maas-api-e2e-api-b-808361-5d568d696b-xsjm9\" (UID: \"9d29605a-ac0f-4068-b11b-fec653825bee\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-808361-5d568d696b-xsjm9"
Jul 14 12:16:48.255929 ip-10-0-136-173 kubenswrapper[2551]: E0714 12:16:48.255920    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d29605a-ac0f-4068-b11b-fec653825bee-maas-api-tls podName:9d29605a-ac0f-4068-b11b-fec653825bee nodeName:}" failed. No retries permitted until 2026-07-14 12:16:48.755899385 +0000 UTC m=+2359.987708663 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9d29605a-ac0f-4068-b11b-fec653825bee-maas-api-tls") pod "maas-api-e2e-api-b-808361-5d568d696b-xsjm9" (UID: "9d29605a-ac0f-4068-b11b-fec653825bee") : secret "maas-api-e2e-api-b-808361-serving-cert" not found
Jul 14 12:16:48.256107 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:48.256002    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jtgg\" (UniqueName: \"kubernetes.io/projected/9d29605a-ac0f-4068-b11b-fec653825bee-kube-api-access-6jtgg\") pod \"maas-api-e2e-api-b-808361-5d568d696b-xsjm9\" (UID: \"9d29605a-ac0f-4068-b11b-fec653825bee\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-808361-5d568d696b-xsjm9"
Jul 14 12:16:48.256430 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:48.256405    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d29605a-ac0f-4068-b11b-fec653825bee-openshift-service-ca\") pod \"maas-api-e2e-api-b-808361-5d568d696b-xsjm9\" (UID: \"9d29605a-ac0f-4068-b11b-fec653825bee\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-808361-5d568d696b-xsjm9"
Jul 14 12:16:48.278063 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:48.278021    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jtgg\" (UniqueName: \"kubernetes.io/projected/9d29605a-ac0f-4068-b11b-fec653825bee-kube-api-access-6jtgg\") pod \"maas-api-e2e-api-b-808361-5d568d696b-xsjm9\" (UID: \"9d29605a-ac0f-4068-b11b-fec653825bee\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-808361-5d568d696b-xsjm9"
Jul 14 12:16:48.345507 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:48.345473    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:16:48.617297 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:48.617258    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:16:48.760820 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:48.760784    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d29605a-ac0f-4068-b11b-fec653825bee-maas-api-tls\") pod \"maas-api-e2e-api-b-808361-5d568d696b-xsjm9\" (UID: \"9d29605a-ac0f-4068-b11b-fec653825bee\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-808361-5d568d696b-xsjm9"
Jul 14 12:16:48.763402 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:48.763373    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d29605a-ac0f-4068-b11b-fec653825bee-maas-api-tls\") pod \"maas-api-e2e-api-b-808361-5d568d696b-xsjm9\" (UID: \"9d29605a-ac0f-4068-b11b-fec653825bee\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-808361-5d568d696b-xsjm9"
Jul 14 12:16:48.827039 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:48.826999    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:16:48.945103 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:48.945067    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-808361-5d568d696b-xsjm9"
Jul 14 12:16:48.972446 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:48.972404    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:16:49.101053 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:49.101028    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-808361-5d568d696b-xsjm9"]
Jul 14 12:16:49.103625 ip-10-0-136-173 kubenswrapper[2551]: W0714 12:16:49.103596    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d29605a_ac0f_4068_b11b_fec653825bee.slice/crio-06b7cb55bf9e4986ca8be5d8acc3edccb4842a6f159c4f5ef2e68364426162d8 WatchSource:0}: Error finding container 06b7cb55bf9e4986ca8be5d8acc3edccb4842a6f159c4f5ef2e68364426162d8: Status 404 returned error can't find the container with id 06b7cb55bf9e4986ca8be5d8acc3edccb4842a6f159c4f5ef2e68364426162d8
Jul 14 12:16:49.855218 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:49.855186    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-808361-5d568d696b-xsjm9" event={"ID":"9d29605a-ac0f-4068-b11b-fec653825bee","Type":"ContainerStarted","Data":"06b7cb55bf9e4986ca8be5d8acc3edccb4842a6f159c4f5ef2e68364426162d8"}
Jul 14 12:16:50.862074 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:50.862032    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-808361-5d568d696b-xsjm9" event={"ID":"9d29605a-ac0f-4068-b11b-fec653825bee","Type":"ContainerStarted","Data":"ccfd418f2b329193b9ce9e66939c08ebb141df90205341ab0ebeb48decee7520"}
Jul 14 12:16:50.862440 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:50.862103    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-808361-5d568d696b-xsjm9"
Jul 14 12:16:50.886670 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:50.886621    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-808361-5d568d696b-xsjm9" podStartSLOduration=3.194724877 podStartE2EDuration="3.886573047s" totalImagesPullingTime="691.84817ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 12:16:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 12:16:50.885651286 +0000 UTC m=+2362.117460582" watchObservedRunningTime="2026-07-14 12:16:50.886573047 +0000 UTC m=+2362.118382401"
Jul 14 12:16:52.416189 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:52.416155    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:16:56.872771 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:16:56.872737    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-808361-5d568d696b-xsjm9"
Jul 14 12:17:00.911539 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:00.911505    2551 generic.go:356] "Generic (PLEG): container finished" podID="b6a34c02-28ba-4bde-8382-1b7d301285da" containerID="66239962ece95652c4fa8e95fb95e9285a4dfd6dede2b6ec26c1ca7c011d752c" exitCode=0
Jul 14 12:17:00.912021 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:00.911616    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-808361-openshift-default-7cf7d4f7fc-4j6l9" event={"ID":"b6a34c02-28ba-4bde-8382-1b7d301285da","Type":"ContainerDied","Data":"66239962ece95652c4fa8e95fb95e9285a4dfd6dede2b6ec26c1ca7c011d752c"}
Jul 14 12:17:00.912073 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:00.912049    2551 scope.go:117] "RemoveContainer" containerID="66239962ece95652c4fa8e95fb95e9285a4dfd6dede2b6ec26c1ca7c011d752c"
Jul 14 12:17:00.913098 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:00.913064    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 14 12:17:00.913202 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:00.913151    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 14 12:17:00.913274 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:00.913225    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 14 12:17:01.396021 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:01.395967    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:17:01.487891 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:01.487827    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:17:01.837485 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:01.837436    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-808361-openshift-default-7cf7d4f7fc-4j6l9"
Jul 14 12:17:01.917814 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:01.917779    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-808361-openshift-default-7cf7d4f7fc-4j6l9" event={"ID":"b6a34c02-28ba-4bde-8382-1b7d301285da","Type":"ContainerStarted","Data":"3c0af406397ad1e2eb65e9d2c655f32fe7d461227e4ac9b8e7d4f4dd4d8ef006"}
Jul 14 12:17:02.294966 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:02.294919    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-808361-openshift-default-7cf7d4f7fc-4j6l9"
Jul 14 12:17:02.295980 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:02.295953    2551 patch_prober.go:28] interesting pod/e2e-api-b-808361-openshift-default-7cf7d4f7fc-4j6l9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Jul 14 12:17:02.296121 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:02.296013    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-808361-openshift-default-7cf7d4f7fc-4j6l9" podUID="b6a34c02-28ba-4bde-8382-1b7d301285da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Jul 14 12:17:02.306327 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:02.306298    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 12:17:03.295805 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:03.295766    2551 patch_prober.go:28] interesting pod/e2e-api-b-808361-openshift-default-7cf7d4f7fc-4j6l9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Jul 14 12:17:03.296209 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:03.295826    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-808361-openshift-default-7cf7d4f7fc-4j6l9" podUID="b6a34c02-28ba-4bde-8382-1b7d301285da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Jul 14 12:17:04.296131 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:04.296096    2551 patch_prober.go:28] interesting pod/e2e-api-b-808361-openshift-default-7cf7d4f7fc-4j6l9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Jul 14 12:17:04.296534 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:04.296150    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-808361-openshift-default-7cf7d4f7fc-4j6l9" podUID="b6a34c02-28ba-4bde-8382-1b7d301285da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Jul 14 12:17:05.295957 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:05.295912    2551 patch_prober.go:28] interesting pod/e2e-api-b-808361-openshift-default-7cf7d4f7fc-4j6l9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Jul 14 12:17:05.296157 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:05.295977    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-808361-openshift-default-7cf7d4f7fc-4j6l9" podUID="b6a34c02-28ba-4bde-8382-1b7d301285da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Jul 14 12:17:06.299700 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:06.299662    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-808361-openshift-default-7cf7d4f7fc-4j6l9"
Jul 14 12:17:06.300193 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:06.299962    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-808361-openshift-default-7cf7d4f7fc-4j6l9"
Jul 14 12:17:06.300812 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:06.300795    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-808361-openshift-default-7cf7d4f7fc-4j6l9"
Jul 14 12:17:09.788808 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:09.788772    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:17:09.892840 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:09.892809    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:17:10.747849 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:10.747810    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3ce01d-57f9d5b6f9-9pvz5"]
Jul 14 12:17:10.748137 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:10.748104    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3ce01d-57f9d5b6f9-9pvz5" podUID="81507b85-3f90-462b-9ec2-07e6785b8c9a" containerName="maas-api" containerID="cri-o://e0bd1e0e039f00ca981bc9733eff32c158a67d9634b9abbf9967326653d936b1" gracePeriod=30
Jul 14 12:17:10.784039 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:10.783997    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:17:10.972328 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:10.972291    2551 generic.go:356] "Generic (PLEG): container finished" podID="81507b85-3f90-462b-9ec2-07e6785b8c9a" containerID="e0bd1e0e039f00ca981bc9733eff32c158a67d9634b9abbf9967326653d936b1" exitCode=0
Jul 14 12:17:10.972773 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:10.972327    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3ce01d-57f9d5b6f9-9pvz5" event={"ID":"81507b85-3f90-462b-9ec2-07e6785b8c9a","Type":"ContainerDied","Data":"e0bd1e0e039f00ca981bc9733eff32c158a67d9634b9abbf9967326653d936b1"}
Jul 14 12:17:11.024358 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:11.024329    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3ce01d-57f9d5b6f9-9pvz5"
Jul 14 12:17:11.072486 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:11.072448    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81507b85-3f90-462b-9ec2-07e6785b8c9a-maas-api-tls\") pod \"81507b85-3f90-462b-9ec2-07e6785b8c9a\" (UID: \"81507b85-3f90-462b-9ec2-07e6785b8c9a\") "
Jul 14 12:17:11.072687 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:11.072504    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/81507b85-3f90-462b-9ec2-07e6785b8c9a-openshift-service-ca\") pod \"81507b85-3f90-462b-9ec2-07e6785b8c9a\" (UID: \"81507b85-3f90-462b-9ec2-07e6785b8c9a\") "
Jul 14 12:17:11.072687 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:11.072548    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zk62n\" (UniqueName: \"kubernetes.io/projected/81507b85-3f90-462b-9ec2-07e6785b8c9a-kube-api-access-zk62n\") pod \"81507b85-3f90-462b-9ec2-07e6785b8c9a\" (UID: \"81507b85-3f90-462b-9ec2-07e6785b8c9a\") "
Jul 14 12:17:11.072883 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:11.072861    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81507b85-3f90-462b-9ec2-07e6785b8c9a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "81507b85-3f90-462b-9ec2-07e6785b8c9a" (UID: "81507b85-3f90-462b-9ec2-07e6785b8c9a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 12:17:11.074745 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:11.074714    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81507b85-3f90-462b-9ec2-07e6785b8c9a-kube-api-access-zk62n" (OuterVolumeSpecName: "kube-api-access-zk62n") pod "81507b85-3f90-462b-9ec2-07e6785b8c9a" (UID: "81507b85-3f90-462b-9ec2-07e6785b8c9a"). InnerVolumeSpecName "kube-api-access-zk62n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 12:17:11.074865 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:11.074794    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81507b85-3f90-462b-9ec2-07e6785b8c9a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "81507b85-3f90-462b-9ec2-07e6785b8c9a" (UID: "81507b85-3f90-462b-9ec2-07e6785b8c9a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 12:17:11.173698 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:11.173656    2551 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/81507b85-3f90-462b-9ec2-07e6785b8c9a-openshift-service-ca\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 12:17:11.173698 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:11.173691    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zk62n\" (UniqueName: \"kubernetes.io/projected/81507b85-3f90-462b-9ec2-07e6785b8c9a-kube-api-access-zk62n\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 12:17:11.173698 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:11.173703    2551 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81507b85-3f90-462b-9ec2-07e6785b8c9a-maas-api-tls\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 12:17:11.979442 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:11.979398    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3ce01d-57f9d5b6f9-9pvz5" event={"ID":"81507b85-3f90-462b-9ec2-07e6785b8c9a","Type":"ContainerDied","Data":"dcd43e3ca51f93391b86a6ebd99068f59e1d3f43ab3d5fab0c9810f6be4eb23b"}
Jul 14 12:17:11.979910 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:11.979463    2551 scope.go:117] "RemoveContainer" containerID="e0bd1e0e039f00ca981bc9733eff32c158a67d9634b9abbf9967326653d936b1"
Jul 14 12:17:11.979910 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:11.979410    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3ce01d-57f9d5b6f9-9pvz5"
Jul 14 12:17:12.000804 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:12.000768    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3ce01d-57f9d5b6f9-9pvz5"]
Jul 14 12:17:12.006518 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:12.006486    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3ce01d-57f9d5b6f9-9pvz5"]
Jul 14 12:17:13.191243 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:13.191209    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81507b85-3f90-462b-9ec2-07e6785b8c9a" path="/var/lib/kubelet/pods/81507b85-3f90-462b-9ec2-07e6785b8c9a/volumes"
Jul 14 12:17:14.546350 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:14.546300    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 12:17:15.653371 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:15.653337    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:17:22.836081 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:22.836043    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:17:22.934414 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:22.934374    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:17:26.383222 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:26.383181    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:17:26.618322 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:26.618285    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:17:29.251955 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:29.251921    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cp57b_62c98a11-ece4-4497-99e7-0840a7626a56/ovn-acl-logging/0.log"
Jul 14 12:17:29.257719 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:29.257695    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cp57b_62c98a11-ece4-4497-99e7-0840a7626a56/ovn-acl-logging/0.log"
Jul 14 12:17:37.320256 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:37.320223    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-808361-5d568d696b-xsjm9"]
Jul 14 12:17:37.320712 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:37.320437    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-808361-5d568d696b-xsjm9" podUID="9d29605a-ac0f-4068-b11b-fec653825bee" containerName="maas-api" containerID="cri-o://ccfd418f2b329193b9ce9e66939c08ebb141df90205341ab0ebeb48decee7520" gracePeriod=30
Jul 14 12:17:37.374649 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:37.374610    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:17:37.576026 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:37.575935    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-808361-5d568d696b-xsjm9"
Jul 14 12:17:37.727164 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:37.727107    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d29605a-ac0f-4068-b11b-fec653825bee-maas-api-tls\") pod \"9d29605a-ac0f-4068-b11b-fec653825bee\" (UID: \"9d29605a-ac0f-4068-b11b-fec653825bee\") "
Jul 14 12:17:37.727365 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:37.727203    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d29605a-ac0f-4068-b11b-fec653825bee-openshift-service-ca\") pod \"9d29605a-ac0f-4068-b11b-fec653825bee\" (UID: \"9d29605a-ac0f-4068-b11b-fec653825bee\") "
Jul 14 12:17:37.727470 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:37.727441    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6jtgg\" (UniqueName: \"kubernetes.io/projected/9d29605a-ac0f-4068-b11b-fec653825bee-kube-api-access-6jtgg\") pod \"9d29605a-ac0f-4068-b11b-fec653825bee\" (UID: \"9d29605a-ac0f-4068-b11b-fec653825bee\") "
Jul 14 12:17:37.727634 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:37.727607    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d29605a-ac0f-4068-b11b-fec653825bee-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9d29605a-ac0f-4068-b11b-fec653825bee" (UID: "9d29605a-ac0f-4068-b11b-fec653825bee"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 12:17:37.727839 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:37.727817    2551 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d29605a-ac0f-4068-b11b-fec653825bee-openshift-service-ca\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 12:17:37.729397 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:37.729365    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d29605a-ac0f-4068-b11b-fec653825bee-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9d29605a-ac0f-4068-b11b-fec653825bee" (UID: "9d29605a-ac0f-4068-b11b-fec653825bee"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 12:17:37.729502 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:37.729467    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d29605a-ac0f-4068-b11b-fec653825bee-kube-api-access-6jtgg" (OuterVolumeSpecName: "kube-api-access-6jtgg") pod "9d29605a-ac0f-4068-b11b-fec653825bee" (UID: "9d29605a-ac0f-4068-b11b-fec653825bee"). InnerVolumeSpecName "kube-api-access-6jtgg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 12:17:37.829020 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:37.828922    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jtgg\" (UniqueName: \"kubernetes.io/projected/9d29605a-ac0f-4068-b11b-fec653825bee-kube-api-access-6jtgg\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 12:17:37.829020 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:37.828962    2551 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d29605a-ac0f-4068-b11b-fec653825bee-maas-api-tls\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 12:17:38.102455 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:38.102362    2551 generic.go:356] "Generic (PLEG): container finished" podID="9d29605a-ac0f-4068-b11b-fec653825bee" containerID="ccfd418f2b329193b9ce9e66939c08ebb141df90205341ab0ebeb48decee7520" exitCode=0
Jul 14 12:17:38.102455 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:38.102412    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-808361-5d568d696b-xsjm9"
Jul 14 12:17:38.102732 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:38.102410    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-808361-5d568d696b-xsjm9" event={"ID":"9d29605a-ac0f-4068-b11b-fec653825bee","Type":"ContainerDied","Data":"ccfd418f2b329193b9ce9e66939c08ebb141df90205341ab0ebeb48decee7520"}
Jul 14 12:17:38.102732 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:38.102554    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-808361-5d568d696b-xsjm9" event={"ID":"9d29605a-ac0f-4068-b11b-fec653825bee","Type":"ContainerDied","Data":"06b7cb55bf9e4986ca8be5d8acc3edccb4842a6f159c4f5ef2e68364426162d8"}
Jul 14 12:17:38.102732 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:38.102622    2551 scope.go:117] "RemoveContainer" containerID="ccfd418f2b329193b9ce9e66939c08ebb141df90205341ab0ebeb48decee7520"
Jul 14 12:17:38.112295 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:38.112275    2551 scope.go:117] "RemoveContainer" containerID="ccfd418f2b329193b9ce9e66939c08ebb141df90205341ab0ebeb48decee7520"
Jul 14 12:17:38.112631 ip-10-0-136-173 kubenswrapper[2551]: E0714 12:17:38.112610    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccfd418f2b329193b9ce9e66939c08ebb141df90205341ab0ebeb48decee7520\": container with ID starting with ccfd418f2b329193b9ce9e66939c08ebb141df90205341ab0ebeb48decee7520 not found: ID does not exist" containerID="ccfd418f2b329193b9ce9e66939c08ebb141df90205341ab0ebeb48decee7520"
Jul 14 12:17:38.112699 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:38.112648    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccfd418f2b329193b9ce9e66939c08ebb141df90205341ab0ebeb48decee7520"} err="failed to get container status \"ccfd418f2b329193b9ce9e66939c08ebb141df90205341ab0ebeb48decee7520\": rpc error: code = NotFound desc = could not find container \"ccfd418f2b329193b9ce9e66939c08ebb141df90205341ab0ebeb48decee7520\": container with ID starting with ccfd418f2b329193b9ce9e66939c08ebb141df90205341ab0ebeb48decee7520 not found: ID does not exist"
Jul 14 12:17:38.128401 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:38.128366    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-808361-5d568d696b-xsjm9"]
Jul 14 12:17:38.134927 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:38.134902    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-808361-5d568d696b-xsjm9"]
Jul 14 12:17:39.191013 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:39.190979    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d29605a-ac0f-4068-b11b-fec653825bee" path="/var/lib/kubelet/pods/9d29605a-ac0f-4068-b11b-fec653825bee/volumes"
Jul 14 12:17:40.700124 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:40.700087    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 12:17:41.898396 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:41.898348    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:17:48.530865 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:48.530823    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:17:48.622876 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:48.622837    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:17:52.617881 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:52.617788    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-808361-openshift-default-7cf7d4f7fc-4j6l9"]
Jul 14 12:17:52.618316 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:52.618062    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-808361-openshift-default-7cf7d4f7fc-4j6l9" podUID="b6a34c02-28ba-4bde-8382-1b7d301285da" containerName="istio-proxy" containerID="cri-o://3c0af406397ad1e2eb65e9d2c655f32fe7d461227e4ac9b8e7d4f4dd4d8ef006" gracePeriod=30
Jul 14 12:17:52.648810 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:52.648771    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:17:52.933661 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:52.933542    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:17:53.328404 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:53.328369    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:17:53.679609 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:53.679498    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:17:54.101287 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:54.101252    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:17:57.101862 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:57.101820    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:17:57.193689 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:57.193653    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-cac82f-7b45ddd4fc-85j8h"]
Jul 14 12:17:57.194042 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:57.194017    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81507b85-3f90-462b-9ec2-07e6785b8c9a" containerName="maas-api"
Jul 14 12:17:57.194042 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:57.194037    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="81507b85-3f90-462b-9ec2-07e6785b8c9a" containerName="maas-api"
Jul 14 12:17:57.194233 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:57.194051    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d29605a-ac0f-4068-b11b-fec653825bee" containerName="maas-api"
Jul 14 12:17:57.194233 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:57.194060    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d29605a-ac0f-4068-b11b-fec653825bee" containerName="maas-api"
Jul 14 12:17:57.194233 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:57.194141    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="81507b85-3f90-462b-9ec2-07e6785b8c9a" containerName="maas-api"
Jul 14 12:17:57.194233 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:57.194156    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d29605a-ac0f-4068-b11b-fec653825bee" containerName="maas-api"
Jul 14 12:17:57.198541 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:57.198519    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-cac82f-7b45ddd4fc-85j8h"
Jul 14 12:17:57.201343 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:57.201319    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-cac82f-serving-cert\""
Jul 14 12:17:57.201497 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:57.201362    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-62k4x\""
Jul 14 12:17:57.201497 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:57.201389    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 14 12:17:57.202563 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:57.202547    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 14 12:17:57.227356 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:57.227320    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-cac82f-7b45ddd4fc-85j8h"]
Jul 14 12:17:57.252438 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:57.252411    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:17:57.289839 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:57.289789    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8efbff05-6d0b-4f2a-be7b-84126004eee8-maas-api-tls\") pod \"maas-api-e2e-shared-a-cac82f-7b45ddd4fc-85j8h\" (UID: \"8efbff05-6d0b-4f2a-be7b-84126004eee8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-cac82f-7b45ddd4fc-85j8h"
Jul 14 12:17:57.290035 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:57.289862    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8swsv\" (UniqueName: \"kubernetes.io/projected/8efbff05-6d0b-4f2a-be7b-84126004eee8-kube-api-access-8swsv\") pod \"maas-api-e2e-shared-a-cac82f-7b45ddd4fc-85j8h\" (UID: \"8efbff05-6d0b-4f2a-be7b-84126004eee8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-cac82f-7b45ddd4fc-85j8h"
Jul 14 12:17:57.290035 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:57.289921    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8efbff05-6d0b-4f2a-be7b-84126004eee8-openshift-service-ca\") pod \"maas-api-e2e-shared-a-cac82f-7b45ddd4fc-85j8h\" (UID: \"8efbff05-6d0b-4f2a-be7b-84126004eee8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-cac82f-7b45ddd4fc-85j8h"
Jul 14 12:17:57.365712 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:57.365572    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:17:57.391018 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:57.390976    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8efbff05-6d0b-4f2a-be7b-84126004eee8-maas-api-tls\") pod \"maas-api-e2e-shared-a-cac82f-7b45ddd4fc-85j8h\" (UID: \"8efbff05-6d0b-4f2a-be7b-84126004eee8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-cac82f-7b45ddd4fc-85j8h"
Jul 14 12:17:57.391252 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:57.391039    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8swsv\" (UniqueName: \"kubernetes.io/projected/8efbff05-6d0b-4f2a-be7b-84126004eee8-kube-api-access-8swsv\") pod \"maas-api-e2e-shared-a-cac82f-7b45ddd4fc-85j8h\" (UID: \"8efbff05-6d0b-4f2a-be7b-84126004eee8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-cac82f-7b45ddd4fc-85j8h"
Jul 14 12:17:57.391252 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:57.391089    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8efbff05-6d0b-4f2a-be7b-84126004eee8-openshift-service-ca\") pod \"maas-api-e2e-shared-a-cac82f-7b45ddd4fc-85j8h\" (UID: \"8efbff05-6d0b-4f2a-be7b-84126004eee8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-cac82f-7b45ddd4fc-85j8h"
Jul 14 12:17:57.391914 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:57.391890    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8efbff05-6d0b-4f2a-be7b-84126004eee8-openshift-service-ca\") pod \"maas-api-e2e-shared-a-cac82f-7b45ddd4fc-85j8h\" (UID: \"8efbff05-6d0b-4f2a-be7b-84126004eee8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-cac82f-7b45ddd4fc-85j8h"
Jul 14 12:17:57.393592 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:57.393556    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8efbff05-6d0b-4f2a-be7b-84126004eee8-maas-api-tls\") pod \"maas-api-e2e-shared-a-cac82f-7b45ddd4fc-85j8h\" (UID: \"8efbff05-6d0b-4f2a-be7b-84126004eee8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-cac82f-7b45ddd4fc-85j8h"
Jul 14 12:17:57.409287 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:57.409251    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8swsv\" (UniqueName: \"kubernetes.io/projected/8efbff05-6d0b-4f2a-be7b-84126004eee8-kube-api-access-8swsv\") pod \"maas-api-e2e-shared-a-cac82f-7b45ddd4fc-85j8h\" (UID: \"8efbff05-6d0b-4f2a-be7b-84126004eee8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-cac82f-7b45ddd4fc-85j8h"
Jul 14 12:17:57.509142 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:57.509095    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-cac82f-7b45ddd4fc-85j8h"
Jul 14 12:17:57.657590 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:57.657553    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-cac82f-7b45ddd4fc-85j8h"]
Jul 14 12:17:57.693729 ip-10-0-136-173 kubenswrapper[2551]: W0714 12:17:57.693690    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8efbff05_6d0b_4f2a_be7b_84126004eee8.slice/crio-c78a905f505932cddf3a22a6a36d909e569e68a9378f377fa2d356dbe7a7eb32 WatchSource:0}: Error finding container c78a905f505932cddf3a22a6a36d909e569e68a9378f377fa2d356dbe7a7eb32: Status 404 returned error can't find the container with id c78a905f505932cddf3a22a6a36d909e569e68a9378f377fa2d356dbe7a7eb32
Jul 14 12:17:57.888449 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:57.888383    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-808361-openshift-default-7cf7d4f7fc-4j6l9"
Jul 14 12:17:57.995439 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:57.995397    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6a34c02-28ba-4bde-8382-1b7d301285da-istio-token\") pod \"b6a34c02-28ba-4bde-8382-1b7d301285da\" (UID: \"b6a34c02-28ba-4bde-8382-1b7d301285da\") "
Jul 14 12:17:57.995439 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:57.995435    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5zfmp\" (UniqueName: \"kubernetes.io/projected/b6a34c02-28ba-4bde-8382-1b7d301285da-kube-api-access-5zfmp\") pod \"b6a34c02-28ba-4bde-8382-1b7d301285da\" (UID: \"b6a34c02-28ba-4bde-8382-1b7d301285da\") "
Jul 14 12:17:57.995744 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:57.995483    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6a34c02-28ba-4bde-8382-1b7d301285da-workload-socket\") pod \"b6a34c02-28ba-4bde-8382-1b7d301285da\" (UID: \"b6a34c02-28ba-4bde-8382-1b7d301285da\") "
Jul 14 12:17:57.995744 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:57.995502    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6a34c02-28ba-4bde-8382-1b7d301285da-istio-data\") pod \"b6a34c02-28ba-4bde-8382-1b7d301285da\" (UID: \"b6a34c02-28ba-4bde-8382-1b7d301285da\") "
Jul 14 12:17:57.995744 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:57.995520    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6a34c02-28ba-4bde-8382-1b7d301285da-istio-podinfo\") pod \"b6a34c02-28ba-4bde-8382-1b7d301285da\" (UID: \"b6a34c02-28ba-4bde-8382-1b7d301285da\") "
Jul 14 12:17:57.995744 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:57.995551    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6a34c02-28ba-4bde-8382-1b7d301285da-istio-envoy\") pod \"b6a34c02-28ba-4bde-8382-1b7d301285da\" (UID: \"b6a34c02-28ba-4bde-8382-1b7d301285da\") "
Jul 14 12:17:57.995744 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:57.995642    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6a34c02-28ba-4bde-8382-1b7d301285da-workload-certs\") pod \"b6a34c02-28ba-4bde-8382-1b7d301285da\" (UID: \"b6a34c02-28ba-4bde-8382-1b7d301285da\") "
Jul 14 12:17:57.995744 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:57.995716    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6a34c02-28ba-4bde-8382-1b7d301285da-istiod-ca-cert\") pod \"b6a34c02-28ba-4bde-8382-1b7d301285da\" (UID: \"b6a34c02-28ba-4bde-8382-1b7d301285da\") "
Jul 14 12:17:57.996117 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:57.995747    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6a34c02-28ba-4bde-8382-1b7d301285da-credential-socket\") pod \"b6a34c02-28ba-4bde-8382-1b7d301285da\" (UID: \"b6a34c02-28ba-4bde-8382-1b7d301285da\") "
Jul 14 12:17:57.996200 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:57.995814    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6a34c02-28ba-4bde-8382-1b7d301285da-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b6a34c02-28ba-4bde-8382-1b7d301285da" (UID: "b6a34c02-28ba-4bde-8382-1b7d301285da"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 12:17:57.996266 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:57.995962    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6a34c02-28ba-4bde-8382-1b7d301285da-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b6a34c02-28ba-4bde-8382-1b7d301285da" (UID: "b6a34c02-28ba-4bde-8382-1b7d301285da"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 12:17:57.996266 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:57.996084    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6a34c02-28ba-4bde-8382-1b7d301285da-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b6a34c02-28ba-4bde-8382-1b7d301285da" (UID: "b6a34c02-28ba-4bde-8382-1b7d301285da"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 12:17:57.996266 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:57.996092    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6a34c02-28ba-4bde-8382-1b7d301285da-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b6a34c02-28ba-4bde-8382-1b7d301285da" (UID: "b6a34c02-28ba-4bde-8382-1b7d301285da"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 12:17:57.996486 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:57.996466    2551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6a34c02-28ba-4bde-8382-1b7d301285da-workload-socket\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 12:17:57.996571 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:57.996489    2551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6a34c02-28ba-4bde-8382-1b7d301285da-workload-certs\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 12:17:57.996571 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:57.996502    2551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6a34c02-28ba-4bde-8382-1b7d301285da-istiod-ca-cert\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 12:17:57.996571 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:57.996514    2551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6a34c02-28ba-4bde-8382-1b7d301285da-credential-socket\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 12:17:57.996769 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:57.996616    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6a34c02-28ba-4bde-8382-1b7d301285da-istio-data" (OuterVolumeSpecName: "istio-data") pod "b6a34c02-28ba-4bde-8382-1b7d301285da" (UID: "b6a34c02-28ba-4bde-8382-1b7d301285da"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 12:17:57.997757 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:57.997735    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6a34c02-28ba-4bde-8382-1b7d301285da-kube-api-access-5zfmp" (OuterVolumeSpecName: "kube-api-access-5zfmp") pod "b6a34c02-28ba-4bde-8382-1b7d301285da" (UID: "b6a34c02-28ba-4bde-8382-1b7d301285da"). InnerVolumeSpecName "kube-api-access-5zfmp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 12:17:57.997883 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:57.997868    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b6a34c02-28ba-4bde-8382-1b7d301285da-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b6a34c02-28ba-4bde-8382-1b7d301285da" (UID: "b6a34c02-28ba-4bde-8382-1b7d301285da"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 14 12:17:57.998001 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:57.997987    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6a34c02-28ba-4bde-8382-1b7d301285da-istio-token" (OuterVolumeSpecName: "istio-token") pod "b6a34c02-28ba-4bde-8382-1b7d301285da" (UID: "b6a34c02-28ba-4bde-8382-1b7d301285da"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 12:17:57.998065 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:57.998044    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6a34c02-28ba-4bde-8382-1b7d301285da-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b6a34c02-28ba-4bde-8382-1b7d301285da" (UID: "b6a34c02-28ba-4bde-8382-1b7d301285da"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 12:17:58.097778 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:58.097732    2551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6a34c02-28ba-4bde-8382-1b7d301285da-istio-token\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 12:17:58.097778 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:58.097772    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zfmp\" (UniqueName: \"kubernetes.io/projected/b6a34c02-28ba-4bde-8382-1b7d301285da-kube-api-access-5zfmp\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 12:17:58.097778 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:58.097787    2551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6a34c02-28ba-4bde-8382-1b7d301285da-istio-data\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 12:17:58.098357 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:58.097799    2551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6a34c02-28ba-4bde-8382-1b7d301285da-istio-podinfo\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 12:17:58.098357 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:58.097810    2551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6a34c02-28ba-4bde-8382-1b7d301285da-istio-envoy\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 12:17:58.145635 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:58.145533    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:17:58.199159 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:58.199122    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-cac82f-7b45ddd4fc-85j8h" event={"ID":"8efbff05-6d0b-4f2a-be7b-84126004eee8","Type":"ContainerStarted","Data":"c78a905f505932cddf3a22a6a36d909e569e68a9378f377fa2d356dbe7a7eb32"}
Jul 14 12:17:58.200723 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:58.200682    2551 generic.go:356] "Generic (PLEG): container finished" podID="b6a34c02-28ba-4bde-8382-1b7d301285da" containerID="3c0af406397ad1e2eb65e9d2c655f32fe7d461227e4ac9b8e7d4f4dd4d8ef006" exitCode=0
Jul 14 12:17:58.200867 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:58.200731    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-808361-openshift-default-7cf7d4f7fc-4j6l9"
Jul 14 12:17:58.200867 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:58.200740    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-808361-openshift-default-7cf7d4f7fc-4j6l9" event={"ID":"b6a34c02-28ba-4bde-8382-1b7d301285da","Type":"ContainerDied","Data":"3c0af406397ad1e2eb65e9d2c655f32fe7d461227e4ac9b8e7d4f4dd4d8ef006"}
Jul 14 12:17:58.200867 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:58.200780    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-808361-openshift-default-7cf7d4f7fc-4j6l9" event={"ID":"b6a34c02-28ba-4bde-8382-1b7d301285da","Type":"ContainerDied","Data":"f7d07da03e3577da17c340f2491cf743e14ba64c6dc33f11978f6bc6c57fda43"}
Jul 14 12:17:58.200867 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:58.200805    2551 scope.go:117] "RemoveContainer" containerID="3c0af406397ad1e2eb65e9d2c655f32fe7d461227e4ac9b8e7d4f4dd4d8ef006"
Jul 14 12:17:58.210086 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:58.210066    2551 scope.go:117] "RemoveContainer" containerID="66239962ece95652c4fa8e95fb95e9285a4dfd6dede2b6ec26c1ca7c011d752c"
Jul 14 12:17:58.218699 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:58.218676    2551 scope.go:117] "RemoveContainer" containerID="3c0af406397ad1e2eb65e9d2c655f32fe7d461227e4ac9b8e7d4f4dd4d8ef006"
Jul 14 12:17:58.219039 ip-10-0-136-173 kubenswrapper[2551]: E0714 12:17:58.219016    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c0af406397ad1e2eb65e9d2c655f32fe7d461227e4ac9b8e7d4f4dd4d8ef006\": container with ID starting with 3c0af406397ad1e2eb65e9d2c655f32fe7d461227e4ac9b8e7d4f4dd4d8ef006 not found: ID does not exist" containerID="3c0af406397ad1e2eb65e9d2c655f32fe7d461227e4ac9b8e7d4f4dd4d8ef006"
Jul 14 12:17:58.219129 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:58.219049    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c0af406397ad1e2eb65e9d2c655f32fe7d461227e4ac9b8e7d4f4dd4d8ef006"} err="failed to get container status \"3c0af406397ad1e2eb65e9d2c655f32fe7d461227e4ac9b8e7d4f4dd4d8ef006\": rpc error: code = NotFound desc = could not find container \"3c0af406397ad1e2eb65e9d2c655f32fe7d461227e4ac9b8e7d4f4dd4d8ef006\": container with ID starting with 3c0af406397ad1e2eb65e9d2c655f32fe7d461227e4ac9b8e7d4f4dd4d8ef006 not found: ID does not exist"
Jul 14 12:17:58.219129 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:58.219075    2551 scope.go:117] "RemoveContainer" containerID="66239962ece95652c4fa8e95fb95e9285a4dfd6dede2b6ec26c1ca7c011d752c"
Jul 14 12:17:58.219371 ip-10-0-136-173 kubenswrapper[2551]: E0714 12:17:58.219350    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66239962ece95652c4fa8e95fb95e9285a4dfd6dede2b6ec26c1ca7c011d752c\": container with ID starting with 66239962ece95652c4fa8e95fb95e9285a4dfd6dede2b6ec26c1ca7c011d752c not found: ID does not exist" containerID="66239962ece95652c4fa8e95fb95e9285a4dfd6dede2b6ec26c1ca7c011d752c"
Jul 14 12:17:58.219431 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:58.219382    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66239962ece95652c4fa8e95fb95e9285a4dfd6dede2b6ec26c1ca7c011d752c"} err="failed to get container status \"66239962ece95652c4fa8e95fb95e9285a4dfd6dede2b6ec26c1ca7c011d752c\": rpc error: code = NotFound desc = could not find container \"66239962ece95652c4fa8e95fb95e9285a4dfd6dede2b6ec26c1ca7c011d752c\": container with ID starting with 66239962ece95652c4fa8e95fb95e9285a4dfd6dede2b6ec26c1ca7c011d752c not found: ID does not exist"
Jul 14 12:17:58.229061 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:58.229030    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-808361-openshift-default-7cf7d4f7fc-4j6l9"]
Jul 14 12:17:58.234654 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:58.234627    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-808361-openshift-default-7cf7d4f7fc-4j6l9"]
Jul 14 12:17:58.236642 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:58.236617    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:17:59.191362 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:59.191324    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6a34c02-28ba-4bde-8382-1b7d301285da" path="/var/lib/kubelet/pods/b6a34c02-28ba-4bde-8382-1b7d301285da/volumes"
Jul 14 12:17:59.207472 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:59.207438    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-cac82f-7b45ddd4fc-85j8h" event={"ID":"8efbff05-6d0b-4f2a-be7b-84126004eee8","Type":"ContainerStarted","Data":"41cf58bd3a7c1cef31206a755bbd7d79647b0c526c60869b36a533ffa89494b4"}
Jul 14 12:17:59.207625 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:59.207572    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-cac82f-7b45ddd4fc-85j8h"
Jul 14 12:17:59.236517 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:17:59.236463    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-cac82f-7b45ddd4fc-85j8h" podStartSLOduration=1.650145186 podStartE2EDuration="2.236445426s" totalImagesPullingTime="586.30024ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 12:17:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 12:17:59.233878922 +0000 UTC m=+2430.465688239" watchObservedRunningTime="2026-07-14 12:17:59.236445426 +0000 UTC m=+2430.468254722"
Jul 14 12:18:01.206141 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:01.206102    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 12:18:01.579464 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:01.579372    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:18:01.917722 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:01.917629    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:18:02.274120 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:02.274086    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:18:02.793364 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:02.793329    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:18:05.217939 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:05.217904    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-cac82f-7b45ddd4fc-85j8h"
Jul 14 12:18:05.268460 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:05.268398    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:18:05.350348 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:05.350312    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-480a7f-5f4888d679-8vzkj"]
Jul 14 12:18:05.350723 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:05.350708    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6a34c02-28ba-4bde-8382-1b7d301285da" containerName="istio-proxy"
Jul 14 12:18:05.350723 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:05.350723    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6a34c02-28ba-4bde-8382-1b7d301285da" containerName="istio-proxy"
Jul 14 12:18:05.350921 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:05.350786    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6a34c02-28ba-4bde-8382-1b7d301285da" containerName="istio-proxy"
Jul 14 12:18:05.350921 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:05.350794    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6a34c02-28ba-4bde-8382-1b7d301285da" containerName="istio-proxy"
Jul 14 12:18:05.354923 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:05.354900    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-480a7f-5f4888d679-8vzkj"
Jul 14 12:18:05.357603 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:05.357566    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-480a7f-serving-cert\""
Jul 14 12:18:05.364451 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:05.364352    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-480a7f-5f4888d679-8vzkj"]
Jul 14 12:18:05.373071 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:05.373020    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:18:05.465211 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:05.465175    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91892839-6603-4e49-a3b3-6ab94ca8ae17-openshift-service-ca\") pod \"maas-api-e2e-shared-b-480a7f-5f4888d679-8vzkj\" (UID: \"91892839-6603-4e49-a3b3-6ab94ca8ae17\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-480a7f-5f4888d679-8vzkj"
Jul 14 12:18:05.465211 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:05.465222    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5lc2\" (UniqueName: \"kubernetes.io/projected/91892839-6603-4e49-a3b3-6ab94ca8ae17-kube-api-access-k5lc2\") pod \"maas-api-e2e-shared-b-480a7f-5f4888d679-8vzkj\" (UID: \"91892839-6603-4e49-a3b3-6ab94ca8ae17\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-480a7f-5f4888d679-8vzkj"
Jul 14 12:18:05.465480 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:05.465335    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91892839-6603-4e49-a3b3-6ab94ca8ae17-maas-api-tls\") pod \"maas-api-e2e-shared-b-480a7f-5f4888d679-8vzkj\" (UID: \"91892839-6603-4e49-a3b3-6ab94ca8ae17\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-480a7f-5f4888d679-8vzkj"
Jul 14 12:18:05.474717 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:05.474607    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:18:05.565800 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:05.565757    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91892839-6603-4e49-a3b3-6ab94ca8ae17-openshift-service-ca\") pod \"maas-api-e2e-shared-b-480a7f-5f4888d679-8vzkj\" (UID: \"91892839-6603-4e49-a3b3-6ab94ca8ae17\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-480a7f-5f4888d679-8vzkj"
Jul 14 12:18:05.565800 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:05.565802    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5lc2\" (UniqueName: \"kubernetes.io/projected/91892839-6603-4e49-a3b3-6ab94ca8ae17-kube-api-access-k5lc2\") pod \"maas-api-e2e-shared-b-480a7f-5f4888d679-8vzkj\" (UID: \"91892839-6603-4e49-a3b3-6ab94ca8ae17\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-480a7f-5f4888d679-8vzkj"
Jul 14 12:18:05.566014 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:05.565857    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91892839-6603-4e49-a3b3-6ab94ca8ae17-maas-api-tls\") pod \"maas-api-e2e-shared-b-480a7f-5f4888d679-8vzkj\" (UID: \"91892839-6603-4e49-a3b3-6ab94ca8ae17\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-480a7f-5f4888d679-8vzkj"
Jul 14 12:18:05.566014 ip-10-0-136-173 kubenswrapper[2551]: E0714 12:18:05.565954    2551 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-480a7f-serving-cert: secret "maas-api-e2e-shared-b-480a7f-serving-cert" not found
Jul 14 12:18:05.566014 ip-10-0-136-173 kubenswrapper[2551]: E0714 12:18:05.566011    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91892839-6603-4e49-a3b3-6ab94ca8ae17-maas-api-tls podName:91892839-6603-4e49-a3b3-6ab94ca8ae17 nodeName:}" failed. No retries permitted until 2026-07-14 12:18:06.065993251 +0000 UTC m=+2437.297802525 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/91892839-6603-4e49-a3b3-6ab94ca8ae17-maas-api-tls") pod "maas-api-e2e-shared-b-480a7f-5f4888d679-8vzkj" (UID: "91892839-6603-4e49-a3b3-6ab94ca8ae17") : secret "maas-api-e2e-shared-b-480a7f-serving-cert" not found
Jul 14 12:18:05.566407 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:05.566387    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91892839-6603-4e49-a3b3-6ab94ca8ae17-openshift-service-ca\") pod \"maas-api-e2e-shared-b-480a7f-5f4888d679-8vzkj\" (UID: \"91892839-6603-4e49-a3b3-6ab94ca8ae17\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-480a7f-5f4888d679-8vzkj"
Jul 14 12:18:05.576729 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:05.576699    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5lc2\" (UniqueName: \"kubernetes.io/projected/91892839-6603-4e49-a3b3-6ab94ca8ae17-kube-api-access-k5lc2\") pod \"maas-api-e2e-shared-b-480a7f-5f4888d679-8vzkj\" (UID: \"91892839-6603-4e49-a3b3-6ab94ca8ae17\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-480a7f-5f4888d679-8vzkj"
Jul 14 12:18:05.951626 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:05.951572    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:18:06.049624 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:06.049552    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:18:06.069796 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:06.069762    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91892839-6603-4e49-a3b3-6ab94ca8ae17-maas-api-tls\") pod \"maas-api-e2e-shared-b-480a7f-5f4888d679-8vzkj\" (UID: \"91892839-6603-4e49-a3b3-6ab94ca8ae17\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-480a7f-5f4888d679-8vzkj"
Jul 14 12:18:06.072390 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:06.072355    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91892839-6603-4e49-a3b3-6ab94ca8ae17-maas-api-tls\") pod \"maas-api-e2e-shared-b-480a7f-5f4888d679-8vzkj\" (UID: \"91892839-6603-4e49-a3b3-6ab94ca8ae17\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-480a7f-5f4888d679-8vzkj"
Jul 14 12:18:06.267893 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:06.267794    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-480a7f-5f4888d679-8vzkj"
Jul 14 12:18:06.415742 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:06.415532    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-480a7f-5f4888d679-8vzkj"]
Jul 14 12:18:06.418514 ip-10-0-136-173 kubenswrapper[2551]: W0714 12:18:06.418479    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91892839_6603_4e49_a3b3_6ab94ca8ae17.slice/crio-afc17bb6656956d71110449e07d5512e4f962df42f2d96a8eb1ee937d0e0257c WatchSource:0}: Error finding container afc17bb6656956d71110449e07d5512e4f962df42f2d96a8eb1ee937d0e0257c: Status 404 returned error can't find the container with id afc17bb6656956d71110449e07d5512e4f962df42f2d96a8eb1ee937d0e0257c
Jul 14 12:18:07.248588 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:07.248485    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-480a7f-5f4888d679-8vzkj" event={"ID":"91892839-6603-4e49-a3b3-6ab94ca8ae17","Type":"ContainerStarted","Data":"6cac5c700384e34064250de15ba580956acfb3a27b1435d97f363072798dbe1b"}
Jul 14 12:18:07.248588 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:07.248538    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-480a7f-5f4888d679-8vzkj" event={"ID":"91892839-6603-4e49-a3b3-6ab94ca8ae17","Type":"ContainerStarted","Data":"afc17bb6656956d71110449e07d5512e4f962df42f2d96a8eb1ee937d0e0257c"}
Jul 14 12:18:07.248588 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:07.248566    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-480a7f-5f4888d679-8vzkj"
Jul 14 12:18:07.273319 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:07.273266    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-480a7f-5f4888d679-8vzkj" podStartSLOduration=1.727889218 podStartE2EDuration="2.273247414s" totalImagesPullingTime="545.358196ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 12:18:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 12:18:07.26965386 +0000 UTC m=+2438.501463155" watchObservedRunningTime="2026-07-14 12:18:07.273247414 +0000 UTC m=+2438.505056709"
Jul 14 12:18:09.685890 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:09.685857    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:18:13.259315 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:13.259279    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-480a7f-5f4888d679-8vzkj"
Jul 14 12:18:16.290046 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:16.290005    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 12:18:19.157242 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:19.157210    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:18:19.263326 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:19.263276    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:18:26.998052 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:26.998007    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:18:27.116215 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:27.116169    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:18:29.380783 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:29.380739    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-cac82f/auth-test-model-cac82f-kserve-7b4cbcc647-54ghx"]
Jul 14 12:18:29.381280 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:29.381264    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6a34c02-28ba-4bde-8382-1b7d301285da" containerName="istio-proxy"
Jul 14 12:18:29.381325 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:29.381284    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6a34c02-28ba-4bde-8382-1b7d301285da" containerName="istio-proxy"
Jul 14 12:18:29.385814 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:29.385782    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-cac82f/auth-test-model-cac82f-kserve-7b4cbcc647-54ghx"
Jul 14 12:18:29.390029 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:29.389991    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-cac82f\"/\"openshift-service-ca.crt\""
Jul 14 12:18:29.390029 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:29.390013    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-cac82f\"/\"default-dockercfg-px7hg\""
Jul 14 12:18:29.390247 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:29.390036    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-cac82f\"/\"kube-root-ca.crt\""
Jul 14 12:18:29.390247 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:29.390141    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-cac82f\"/\"auth-test-model-cac82f-kserve-self-signed-certs\""
Jul 14 12:18:29.394624 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:29.394565    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-cac82f/auth-test-model-cac82f-kserve-7b4cbcc647-54ghx"]
Jul 14 12:18:29.425423 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:29.425386    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:18:29.483425 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:29.483386    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8855a412-12cd-4c5b-a542-c7374fcbec62-home\") pod \"auth-test-model-cac82f-kserve-7b4cbcc647-54ghx\" (UID: \"8855a412-12cd-4c5b-a542-c7374fcbec62\") " pod="ai-tenant-e2e-shared-a-cac82f/auth-test-model-cac82f-kserve-7b4cbcc647-54ghx"
Jul 14 12:18:29.483425 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:29.483426    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8855a412-12cd-4c5b-a542-c7374fcbec62-tls-certs\") pod \"auth-test-model-cac82f-kserve-7b4cbcc647-54ghx\" (UID: \"8855a412-12cd-4c5b-a542-c7374fcbec62\") " pod="ai-tenant-e2e-shared-a-cac82f/auth-test-model-cac82f-kserve-7b4cbcc647-54ghx"
Jul 14 12:18:29.483750 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:29.483461    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8855a412-12cd-4c5b-a542-c7374fcbec62-tmp-dir\") pod \"auth-test-model-cac82f-kserve-7b4cbcc647-54ghx\" (UID: \"8855a412-12cd-4c5b-a542-c7374fcbec62\") " pod="ai-tenant-e2e-shared-a-cac82f/auth-test-model-cac82f-kserve-7b4cbcc647-54ghx"
Jul 14 12:18:29.483750 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:29.483509    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5x7d\" (UniqueName: \"kubernetes.io/projected/8855a412-12cd-4c5b-a542-c7374fcbec62-kube-api-access-c5x7d\") pod \"auth-test-model-cac82f-kserve-7b4cbcc647-54ghx\" (UID: \"8855a412-12cd-4c5b-a542-c7374fcbec62\") " pod="ai-tenant-e2e-shared-a-cac82f/auth-test-model-cac82f-kserve-7b4cbcc647-54ghx"
Jul 14 12:18:29.483750 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:29.483545    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8855a412-12cd-4c5b-a542-c7374fcbec62-dshm\") pod \"auth-test-model-cac82f-kserve-7b4cbcc647-54ghx\" (UID: \"8855a412-12cd-4c5b-a542-c7374fcbec62\") " pod="ai-tenant-e2e-shared-a-cac82f/auth-test-model-cac82f-kserve-7b4cbcc647-54ghx"
Jul 14 12:18:29.483750 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:29.483684    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8855a412-12cd-4c5b-a542-c7374fcbec62-model-cache\") pod \"auth-test-model-cac82f-kserve-7b4cbcc647-54ghx\" (UID: \"8855a412-12cd-4c5b-a542-c7374fcbec62\") " pod="ai-tenant-e2e-shared-a-cac82f/auth-test-model-cac82f-kserve-7b4cbcc647-54ghx"
Jul 14 12:18:29.584252 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:29.584199    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8855a412-12cd-4c5b-a542-c7374fcbec62-model-cache\") pod \"auth-test-model-cac82f-kserve-7b4cbcc647-54ghx\" (UID: \"8855a412-12cd-4c5b-a542-c7374fcbec62\") " pod="ai-tenant-e2e-shared-a-cac82f/auth-test-model-cac82f-kserve-7b4cbcc647-54ghx"
Jul 14 12:18:29.584252 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:29.584258    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8855a412-12cd-4c5b-a542-c7374fcbec62-home\") pod \"auth-test-model-cac82f-kserve-7b4cbcc647-54ghx\" (UID: \"8855a412-12cd-4c5b-a542-c7374fcbec62\") " pod="ai-tenant-e2e-shared-a-cac82f/auth-test-model-cac82f-kserve-7b4cbcc647-54ghx"
Jul 14 12:18:29.584534 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:29.584285    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8855a412-12cd-4c5b-a542-c7374fcbec62-tls-certs\") pod \"auth-test-model-cac82f-kserve-7b4cbcc647-54ghx\" (UID: \"8855a412-12cd-4c5b-a542-c7374fcbec62\") " pod="ai-tenant-e2e-shared-a-cac82f/auth-test-model-cac82f-kserve-7b4cbcc647-54ghx"
Jul 14 12:18:29.584534 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:29.584325    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8855a412-12cd-4c5b-a542-c7374fcbec62-tmp-dir\") pod \"auth-test-model-cac82f-kserve-7b4cbcc647-54ghx\" (UID: \"8855a412-12cd-4c5b-a542-c7374fcbec62\") " pod="ai-tenant-e2e-shared-a-cac82f/auth-test-model-cac82f-kserve-7b4cbcc647-54ghx"
Jul 14 12:18:29.584534 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:29.584469    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5x7d\" (UniqueName: \"kubernetes.io/projected/8855a412-12cd-4c5b-a542-c7374fcbec62-kube-api-access-c5x7d\") pod \"auth-test-model-cac82f-kserve-7b4cbcc647-54ghx\" (UID: \"8855a412-12cd-4c5b-a542-c7374fcbec62\") " pod="ai-tenant-e2e-shared-a-cac82f/auth-test-model-cac82f-kserve-7b4cbcc647-54ghx"
Jul 14 12:18:29.584534 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:29.584521    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8855a412-12cd-4c5b-a542-c7374fcbec62-dshm\") pod \"auth-test-model-cac82f-kserve-7b4cbcc647-54ghx\" (UID: \"8855a412-12cd-4c5b-a542-c7374fcbec62\") " pod="ai-tenant-e2e-shared-a-cac82f/auth-test-model-cac82f-kserve-7b4cbcc647-54ghx"
Jul 14 12:18:29.584780 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:29.584667    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8855a412-12cd-4c5b-a542-c7374fcbec62-model-cache\") pod \"auth-test-model-cac82f-kserve-7b4cbcc647-54ghx\" (UID: \"8855a412-12cd-4c5b-a542-c7374fcbec62\") " pod="ai-tenant-e2e-shared-a-cac82f/auth-test-model-cac82f-kserve-7b4cbcc647-54ghx"
Jul 14 12:18:29.584780 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:29.584719    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8855a412-12cd-4c5b-a542-c7374fcbec62-tmp-dir\") pod \"auth-test-model-cac82f-kserve-7b4cbcc647-54ghx\" (UID: \"8855a412-12cd-4c5b-a542-c7374fcbec62\") " pod="ai-tenant-e2e-shared-a-cac82f/auth-test-model-cac82f-kserve-7b4cbcc647-54ghx"
Jul 14 12:18:29.584780 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:29.584752    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8855a412-12cd-4c5b-a542-c7374fcbec62-home\") pod \"auth-test-model-cac82f-kserve-7b4cbcc647-54ghx\" (UID: \"8855a412-12cd-4c5b-a542-c7374fcbec62\") " pod="ai-tenant-e2e-shared-a-cac82f/auth-test-model-cac82f-kserve-7b4cbcc647-54ghx"
Jul 14 12:18:29.586737 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:29.586714    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8855a412-12cd-4c5b-a542-c7374fcbec62-dshm\") pod \"auth-test-model-cac82f-kserve-7b4cbcc647-54ghx\" (UID: \"8855a412-12cd-4c5b-a542-c7374fcbec62\") " pod="ai-tenant-e2e-shared-a-cac82f/auth-test-model-cac82f-kserve-7b4cbcc647-54ghx"
Jul 14 12:18:29.586987 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:29.586970    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8855a412-12cd-4c5b-a542-c7374fcbec62-tls-certs\") pod \"auth-test-model-cac82f-kserve-7b4cbcc647-54ghx\" (UID: \"8855a412-12cd-4c5b-a542-c7374fcbec62\") " pod="ai-tenant-e2e-shared-a-cac82f/auth-test-model-cac82f-kserve-7b4cbcc647-54ghx"
Jul 14 12:18:29.595925 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:29.595895    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5x7d\" (UniqueName: \"kubernetes.io/projected/8855a412-12cd-4c5b-a542-c7374fcbec62-kube-api-access-c5x7d\") pod \"auth-test-model-cac82f-kserve-7b4cbcc647-54ghx\" (UID: \"8855a412-12cd-4c5b-a542-c7374fcbec62\") " pod="ai-tenant-e2e-shared-a-cac82f/auth-test-model-cac82f-kserve-7b4cbcc647-54ghx"
Jul 14 12:18:29.700902 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:29.700860    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-cac82f/auth-test-model-cac82f-kserve-7b4cbcc647-54ghx"
Jul 14 12:18:29.849652 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:29.849393    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-cac82f/auth-test-model-cac82f-kserve-7b4cbcc647-54ghx"]
Jul 14 12:18:29.851851 ip-10-0-136-173 kubenswrapper[2551]: W0714 12:18:29.851820    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8855a412_12cd_4c5b_a542_c7374fcbec62.slice/crio-2bb7dd41448138e804fc7788a0e776619f58078dc07e3badf14e9a15d6e2bed6 WatchSource:0}: Error finding container 2bb7dd41448138e804fc7788a0e776619f58078dc07e3badf14e9a15d6e2bed6: Status 404 returned error can't find the container with id 2bb7dd41448138e804fc7788a0e776619f58078dc07e3badf14e9a15d6e2bed6
Jul 14 12:18:30.362655 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:30.362602    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-cac82f/auth-test-model-cac82f-kserve-7b4cbcc647-54ghx" event={"ID":"8855a412-12cd-4c5b-a542-c7374fcbec62","Type":"ContainerStarted","Data":"0d1e78bfdab1ee5fbb9e72bb2e4c8d17e2bf5f5a43be42823d99952c5b3c1433"}
Jul 14 12:18:30.362655 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:30.362668    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-cac82f/auth-test-model-cac82f-kserve-7b4cbcc647-54ghx" event={"ID":"8855a412-12cd-4c5b-a542-c7374fcbec62","Type":"ContainerStarted","Data":"2bb7dd41448138e804fc7788a0e776619f58078dc07e3badf14e9a15d6e2bed6"}
Jul 14 12:18:30.384343 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:30.384290    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-cac82f/auth-test-model-cac82f-kserve-7b4cbcc647-54ghx" podStartSLOduration=1.3842737569999999 podStartE2EDuration="1.384273757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 12:18:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 12:18:30.382914373 +0000 UTC m=+2461.614723696" watchObservedRunningTime="2026-07-14 12:18:30.384273757 +0000 UTC m=+2461.616083053"
Jul 14 12:18:31.506090 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:31.506039    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 12:18:39.701253 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:39.701215    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-cac82f/auth-test-model-cac82f-kserve-7b4cbcc647-54ghx"
Jul 14 12:18:39.701730 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:39.701275    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-cac82f/auth-test-model-cac82f-kserve-7b4cbcc647-54ghx"
Jul 14 12:18:39.714134 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:39.714103    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-cac82f/auth-test-model-cac82f-kserve-7b4cbcc647-54ghx"
Jul 14 12:18:40.421011 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:40.420974    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-cac82f/auth-test-model-cac82f-kserve-7b4cbcc647-54ghx"
Jul 14 12:18:47.242678 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:47.242641    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-480a7f/auth-test-model-480a7f-kserve-54b4749f47-njtpv"]
Jul 14 12:18:47.246131 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:47.246111    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-480a7f/auth-test-model-480a7f-kserve-54b4749f47-njtpv"
Jul 14 12:18:47.267066 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:47.267035    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-480a7f\"/\"openshift-service-ca.crt\""
Jul 14 12:18:47.267280 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:47.267100    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-480a7f\"/\"auth-test-model-480a7f-kserve-self-signed-certs\""
Jul 14 12:18:47.267280 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:47.267132    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-480a7f\"/\"kube-root-ca.crt\""
Jul 14 12:18:47.267280 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:47.267138    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-480a7f\"/\"default-dockercfg-mc2m5\""
Jul 14 12:18:47.290920 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:47.290881    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-480a7f/auth-test-model-480a7f-kserve-54b4749f47-njtpv"]
Jul 14 12:18:47.328946 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:47.328911    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gvbk\" (UniqueName: \"kubernetes.io/projected/863b6c1f-91bc-4103-8d1a-d12b5f265730-kube-api-access-8gvbk\") pod \"auth-test-model-480a7f-kserve-54b4749f47-njtpv\" (UID: \"863b6c1f-91bc-4103-8d1a-d12b5f265730\") " pod="ai-tenant-e2e-shared-b-480a7f/auth-test-model-480a7f-kserve-54b4749f47-njtpv"
Jul 14 12:18:47.329140 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:47.328960    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/863b6c1f-91bc-4103-8d1a-d12b5f265730-model-cache\") pod \"auth-test-model-480a7f-kserve-54b4749f47-njtpv\" (UID: \"863b6c1f-91bc-4103-8d1a-d12b5f265730\") " pod="ai-tenant-e2e-shared-b-480a7f/auth-test-model-480a7f-kserve-54b4749f47-njtpv"
Jul 14 12:18:47.329140 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:47.328982    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/863b6c1f-91bc-4103-8d1a-d12b5f265730-dshm\") pod \"auth-test-model-480a7f-kserve-54b4749f47-njtpv\" (UID: \"863b6c1f-91bc-4103-8d1a-d12b5f265730\") " pod="ai-tenant-e2e-shared-b-480a7f/auth-test-model-480a7f-kserve-54b4749f47-njtpv"
Jul 14 12:18:47.329140 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:47.329052    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/863b6c1f-91bc-4103-8d1a-d12b5f265730-tls-certs\") pod \"auth-test-model-480a7f-kserve-54b4749f47-njtpv\" (UID: \"863b6c1f-91bc-4103-8d1a-d12b5f265730\") " pod="ai-tenant-e2e-shared-b-480a7f/auth-test-model-480a7f-kserve-54b4749f47-njtpv"
Jul 14 12:18:47.329140 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:47.329119    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/863b6c1f-91bc-4103-8d1a-d12b5f265730-home\") pod \"auth-test-model-480a7f-kserve-54b4749f47-njtpv\" (UID: \"863b6c1f-91bc-4103-8d1a-d12b5f265730\") " pod="ai-tenant-e2e-shared-b-480a7f/auth-test-model-480a7f-kserve-54b4749f47-njtpv"
Jul 14 12:18:47.329299 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:47.329192    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/863b6c1f-91bc-4103-8d1a-d12b5f265730-tmp-dir\") pod \"auth-test-model-480a7f-kserve-54b4749f47-njtpv\" (UID: \"863b6c1f-91bc-4103-8d1a-d12b5f265730\") " pod="ai-tenant-e2e-shared-b-480a7f/auth-test-model-480a7f-kserve-54b4749f47-njtpv"
Jul 14 12:18:47.402749 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:47.402709    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:18:47.430356 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:47.430316    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gvbk\" (UniqueName: \"kubernetes.io/projected/863b6c1f-91bc-4103-8d1a-d12b5f265730-kube-api-access-8gvbk\") pod \"auth-test-model-480a7f-kserve-54b4749f47-njtpv\" (UID: \"863b6c1f-91bc-4103-8d1a-d12b5f265730\") " pod="ai-tenant-e2e-shared-b-480a7f/auth-test-model-480a7f-kserve-54b4749f47-njtpv"
Jul 14 12:18:47.430356 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:47.430367    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/863b6c1f-91bc-4103-8d1a-d12b5f265730-model-cache\") pod \"auth-test-model-480a7f-kserve-54b4749f47-njtpv\" (UID: \"863b6c1f-91bc-4103-8d1a-d12b5f265730\") " pod="ai-tenant-e2e-shared-b-480a7f/auth-test-model-480a7f-kserve-54b4749f47-njtpv"
Jul 14 12:18:47.430644 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:47.430389    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/863b6c1f-91bc-4103-8d1a-d12b5f265730-dshm\") pod \"auth-test-model-480a7f-kserve-54b4749f47-njtpv\" (UID: \"863b6c1f-91bc-4103-8d1a-d12b5f265730\") " pod="ai-tenant-e2e-shared-b-480a7f/auth-test-model-480a7f-kserve-54b4749f47-njtpv"
Jul 14 12:18:47.430644 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:47.430414    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/863b6c1f-91bc-4103-8d1a-d12b5f265730-tls-certs\") pod \"auth-test-model-480a7f-kserve-54b4749f47-njtpv\" (UID: \"863b6c1f-91bc-4103-8d1a-d12b5f265730\") " pod="ai-tenant-e2e-shared-b-480a7f/auth-test-model-480a7f-kserve-54b4749f47-njtpv"
Jul 14 12:18:47.430818 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:47.430794    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/863b6c1f-91bc-4103-8d1a-d12b5f265730-model-cache\") pod \"auth-test-model-480a7f-kserve-54b4749f47-njtpv\" (UID: \"863b6c1f-91bc-4103-8d1a-d12b5f265730\") " pod="ai-tenant-e2e-shared-b-480a7f/auth-test-model-480a7f-kserve-54b4749f47-njtpv"
Jul 14 12:18:47.430883 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:47.430808    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/863b6c1f-91bc-4103-8d1a-d12b5f265730-home\") pod \"auth-test-model-480a7f-kserve-54b4749f47-njtpv\" (UID: \"863b6c1f-91bc-4103-8d1a-d12b5f265730\") " pod="ai-tenant-e2e-shared-b-480a7f/auth-test-model-480a7f-kserve-54b4749f47-njtpv"
Jul 14 12:18:47.431064 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:47.430931    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/863b6c1f-91bc-4103-8d1a-d12b5f265730-tmp-dir\") pod \"auth-test-model-480a7f-kserve-54b4749f47-njtpv\" (UID: \"863b6c1f-91bc-4103-8d1a-d12b5f265730\") " pod="ai-tenant-e2e-shared-b-480a7f/auth-test-model-480a7f-kserve-54b4749f47-njtpv"
Jul 14 12:18:47.431220 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:47.431081    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/863b6c1f-91bc-4103-8d1a-d12b5f265730-home\") pod \"auth-test-model-480a7f-kserve-54b4749f47-njtpv\" (UID: \"863b6c1f-91bc-4103-8d1a-d12b5f265730\") " pod="ai-tenant-e2e-shared-b-480a7f/auth-test-model-480a7f-kserve-54b4749f47-njtpv"
Jul 14 12:18:47.431220 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:47.431206    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/863b6c1f-91bc-4103-8d1a-d12b5f265730-tmp-dir\") pod \"auth-test-model-480a7f-kserve-54b4749f47-njtpv\" (UID: \"863b6c1f-91bc-4103-8d1a-d12b5f265730\") " pod="ai-tenant-e2e-shared-b-480a7f/auth-test-model-480a7f-kserve-54b4749f47-njtpv"
Jul 14 12:18:47.433351 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:47.433322    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/863b6c1f-91bc-4103-8d1a-d12b5f265730-dshm\") pod \"auth-test-model-480a7f-kserve-54b4749f47-njtpv\" (UID: \"863b6c1f-91bc-4103-8d1a-d12b5f265730\") " pod="ai-tenant-e2e-shared-b-480a7f/auth-test-model-480a7f-kserve-54b4749f47-njtpv"
Jul 14 12:18:47.433489 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:47.433470    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/863b6c1f-91bc-4103-8d1a-d12b5f265730-tls-certs\") pod \"auth-test-model-480a7f-kserve-54b4749f47-njtpv\" (UID: \"863b6c1f-91bc-4103-8d1a-d12b5f265730\") " pod="ai-tenant-e2e-shared-b-480a7f/auth-test-model-480a7f-kserve-54b4749f47-njtpv"
Jul 14 12:18:47.442983 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:47.442953    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gvbk\" (UniqueName: \"kubernetes.io/projected/863b6c1f-91bc-4103-8d1a-d12b5f265730-kube-api-access-8gvbk\") pod \"auth-test-model-480a7f-kserve-54b4749f47-njtpv\" (UID: \"863b6c1f-91bc-4103-8d1a-d12b5f265730\") " pod="ai-tenant-e2e-shared-b-480a7f/auth-test-model-480a7f-kserve-54b4749f47-njtpv"
Jul 14 12:18:47.556213 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:47.556112    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-480a7f/auth-test-model-480a7f-kserve-54b4749f47-njtpv"
Jul 14 12:18:47.709905 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:47.709876    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-480a7f/auth-test-model-480a7f-kserve-54b4749f47-njtpv"]
Jul 14 12:18:47.712468 ip-10-0-136-173 kubenswrapper[2551]: W0714 12:18:47.712438    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod863b6c1f_91bc_4103_8d1a_d12b5f265730.slice/crio-15f50e55136eed1fbab8770dee084b1326055390990587900ff3e8192654eb27 WatchSource:0}: Error finding container 15f50e55136eed1fbab8770dee084b1326055390990587900ff3e8192654eb27: Status 404 returned error can't find the container with id 15f50e55136eed1fbab8770dee084b1326055390990587900ff3e8192654eb27
Jul 14 12:18:48.451739 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:48.451702    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-480a7f/auth-test-model-480a7f-kserve-54b4749f47-njtpv" event={"ID":"863b6c1f-91bc-4103-8d1a-d12b5f265730","Type":"ContainerStarted","Data":"2091a798edd236af61021521e91352f225ac7c850cb8ea06fe1376c1c0615342"}
Jul 14 12:18:48.451739 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:48.451748    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-480a7f/auth-test-model-480a7f-kserve-54b4749f47-njtpv" event={"ID":"863b6c1f-91bc-4103-8d1a-d12b5f265730","Type":"ContainerStarted","Data":"15f50e55136eed1fbab8770dee084b1326055390990587900ff3e8192654eb27"}
Jul 14 12:18:48.475248 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:48.475190    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-480a7f/auth-test-model-480a7f-kserve-54b4749f47-njtpv" podStartSLOduration=1.475169775 podStartE2EDuration="1.475169775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 12:18:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 12:18:48.473851762 +0000 UTC m=+2479.705661060" watchObservedRunningTime="2026-07-14 12:18:48.475169775 +0000 UTC m=+2479.706979070"
Jul 14 12:18:51.402482 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:51.402442    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 12:18:57.556664 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:57.556629    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-480a7f/auth-test-model-480a7f-kserve-54b4749f47-njtpv"
Jul 14 12:18:57.557080 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:57.556679    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-480a7f/auth-test-model-480a7f-kserve-54b4749f47-njtpv"
Jul 14 12:18:57.569450 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:57.569420    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-480a7f/auth-test-model-480a7f-kserve-54b4749f47-njtpv"
Jul 14 12:18:58.509407 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:18:58.509373    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-480a7f/auth-test-model-480a7f-kserve-54b4749f47-njtpv"
Jul 14 12:19:06.594932 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:19:06.594893    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 12:19:24.098084 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:19:24.097999    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 12:19:42.805835 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:19:42.805790    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 12:19:58.599228 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:19:58.599183    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 12:20:16.495848 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:20:16.495810    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 12:20:36.113834 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:20:36.113795    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 12:20:49.145761 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:20:49.145728    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:20:49.232666 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:20:49.232628    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:20:50.598883 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:20:50.598786    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-cac82f/auth-test-model-cac82f-kserve-7b4cbcc647-54ghx"]
Jul 14 12:20:50.639374 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:20:50.639317    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:20:51.592769 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:20:51.592734    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-480a7f/auth-test-model-480a7f-kserve-54b4749f47-njtpv"]
Jul 14 12:20:51.627439 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:20:51.627401    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:20:54.415655 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:20:54.415611    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 12:20:56.139801 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:20:56.139770    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:21:05.621261 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:21:05.621216    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-cac82f/auth-test-model-cac82f-kserve-7b4cbcc647-54ghx" podUID="8855a412-12cd-4c5b-a542-c7374fcbec62" containerName="main" containerID="cri-o://0d1e78bfdab1ee5fbb9e72bb2e4c8d17e2bf5f5a43be42823d99952c5b3c1433" gracePeriod=45
Jul 14 12:21:05.876028 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:21:05.875952    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-cac82f/auth-test-model-cac82f-kserve-7b4cbcc647-54ghx"
Jul 14 12:21:05.977955 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:21:05.977912    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8855a412-12cd-4c5b-a542-c7374fcbec62-tls-certs\") pod \"8855a412-12cd-4c5b-a542-c7374fcbec62\" (UID: \"8855a412-12cd-4c5b-a542-c7374fcbec62\") "
Jul 14 12:21:05.978118 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:21:05.977973    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8855a412-12cd-4c5b-a542-c7374fcbec62-tmp-dir\") pod \"8855a412-12cd-4c5b-a542-c7374fcbec62\" (UID: \"8855a412-12cd-4c5b-a542-c7374fcbec62\") "
Jul 14 12:21:05.978118 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:21:05.978000    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8855a412-12cd-4c5b-a542-c7374fcbec62-dshm\") pod \"8855a412-12cd-4c5b-a542-c7374fcbec62\" (UID: \"8855a412-12cd-4c5b-a542-c7374fcbec62\") "
Jul 14 12:21:05.978226 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:21:05.978130    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8855a412-12cd-4c5b-a542-c7374fcbec62-home\") pod \"8855a412-12cd-4c5b-a542-c7374fcbec62\" (UID: \"8855a412-12cd-4c5b-a542-c7374fcbec62\") "
Jul 14 12:21:05.978226 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:21:05.978172    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8855a412-12cd-4c5b-a542-c7374fcbec62-model-cache\") pod \"8855a412-12cd-4c5b-a542-c7374fcbec62\" (UID: \"8855a412-12cd-4c5b-a542-c7374fcbec62\") "
Jul 14 12:21:05.978318 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:21:05.978276    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8855a412-12cd-4c5b-a542-c7374fcbec62-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8855a412-12cd-4c5b-a542-c7374fcbec62" (UID: "8855a412-12cd-4c5b-a542-c7374fcbec62"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 12:21:05.978318 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:21:05.978305    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c5x7d\" (UniqueName: \"kubernetes.io/projected/8855a412-12cd-4c5b-a542-c7374fcbec62-kube-api-access-c5x7d\") pod \"8855a412-12cd-4c5b-a542-c7374fcbec62\" (UID: \"8855a412-12cd-4c5b-a542-c7374fcbec62\") "
Jul 14 12:21:05.978425 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:21:05.978372    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8855a412-12cd-4c5b-a542-c7374fcbec62-home" (OuterVolumeSpecName: "home") pod "8855a412-12cd-4c5b-a542-c7374fcbec62" (UID: "8855a412-12cd-4c5b-a542-c7374fcbec62"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 12:21:05.978530 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:21:05.978499    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8855a412-12cd-4c5b-a542-c7374fcbec62-model-cache" (OuterVolumeSpecName: "model-cache") pod "8855a412-12cd-4c5b-a542-c7374fcbec62" (UID: "8855a412-12cd-4c5b-a542-c7374fcbec62"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 12:21:05.978666 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:21:05.978644    2551 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8855a412-12cd-4c5b-a542-c7374fcbec62-tmp-dir\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 12:21:05.978795 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:21:05.978679    2551 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8855a412-12cd-4c5b-a542-c7374fcbec62-home\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 12:21:05.978795 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:21:05.978694    2551 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8855a412-12cd-4c5b-a542-c7374fcbec62-model-cache\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 12:21:05.980303 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:21:05.980274    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8855a412-12cd-4c5b-a542-c7374fcbec62-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8855a412-12cd-4c5b-a542-c7374fcbec62" (UID: "8855a412-12cd-4c5b-a542-c7374fcbec62"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 12:21:05.980303 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:21:05.980276    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8855a412-12cd-4c5b-a542-c7374fcbec62-dshm" (OuterVolumeSpecName: "dshm") pod "8855a412-12cd-4c5b-a542-c7374fcbec62" (UID: "8855a412-12cd-4c5b-a542-c7374fcbec62"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 12:21:05.980467 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:21:05.980335    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8855a412-12cd-4c5b-a542-c7374fcbec62-kube-api-access-c5x7d" (OuterVolumeSpecName: "kube-api-access-c5x7d") pod "8855a412-12cd-4c5b-a542-c7374fcbec62" (UID: "8855a412-12cd-4c5b-a542-c7374fcbec62"). InnerVolumeSpecName "kube-api-access-c5x7d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 12:21:06.079276 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:21:06.079236    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5x7d\" (UniqueName: \"kubernetes.io/projected/8855a412-12cd-4c5b-a542-c7374fcbec62-kube-api-access-c5x7d\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 12:21:06.079276 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:21:06.079271    2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8855a412-12cd-4c5b-a542-c7374fcbec62-tls-certs\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 12:21:06.079276 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:21:06.079281    2551 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8855a412-12cd-4c5b-a542-c7374fcbec62-dshm\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 12:21:06.111472 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:21:06.111433    2551 generic.go:356] "Generic (PLEG): container finished" podID="8855a412-12cd-4c5b-a542-c7374fcbec62" containerID="0d1e78bfdab1ee5fbb9e72bb2e4c8d17e2bf5f5a43be42823d99952c5b3c1433" exitCode=0
Jul 14 12:21:06.111674 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:21:06.111474    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-cac82f/auth-test-model-cac82f-kserve-7b4cbcc647-54ghx"
Jul 14 12:21:06.111674 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:21:06.111524    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-cac82f/auth-test-model-cac82f-kserve-7b4cbcc647-54ghx" event={"ID":"8855a412-12cd-4c5b-a542-c7374fcbec62","Type":"ContainerDied","Data":"0d1e78bfdab1ee5fbb9e72bb2e4c8d17e2bf5f5a43be42823d99952c5b3c1433"}
Jul 14 12:21:06.111674 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:21:06.111561    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-cac82f/auth-test-model-cac82f-kserve-7b4cbcc647-54ghx" event={"ID":"8855a412-12cd-4c5b-a542-c7374fcbec62","Type":"ContainerDied","Data":"2bb7dd41448138e804fc7788a0e776619f58078dc07e3badf14e9a15d6e2bed6"}
Jul 14 12:21:06.111674 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:21:06.111602    2551 scope.go:117] "RemoveContainer" containerID="0d1e78bfdab1ee5fbb9e72bb2e4c8d17e2bf5f5a43be42823d99952c5b3c1433"
Jul 14 12:21:06.123640 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:21:06.123609    2551 scope.go:117] "RemoveContainer" containerID="0d1e78bfdab1ee5fbb9e72bb2e4c8d17e2bf5f5a43be42823d99952c5b3c1433"
Jul 14 12:21:06.124019 ip-10-0-136-173 kubenswrapper[2551]: E0714 12:21:06.123991    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d1e78bfdab1ee5fbb9e72bb2e4c8d17e2bf5f5a43be42823d99952c5b3c1433\": container with ID starting with 0d1e78bfdab1ee5fbb9e72bb2e4c8d17e2bf5f5a43be42823d99952c5b3c1433 not found: ID does not exist" containerID="0d1e78bfdab1ee5fbb9e72bb2e4c8d17e2bf5f5a43be42823d99952c5b3c1433"
Jul 14 12:21:06.124129 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:21:06.124031    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d1e78bfdab1ee5fbb9e72bb2e4c8d17e2bf5f5a43be42823d99952c5b3c1433"} err="failed to get container status \"0d1e78bfdab1ee5fbb9e72bb2e4c8d17e2bf5f5a43be42823d99952c5b3c1433\": rpc error: code = NotFound desc = could not find container \"0d1e78bfdab1ee5fbb9e72bb2e4c8d17e2bf5f5a43be42823d99952c5b3c1433\": container with ID starting with 0d1e78bfdab1ee5fbb9e72bb2e4c8d17e2bf5f5a43be42823d99952c5b3c1433 not found: ID does not exist"
Jul 14 12:21:06.151777 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:21:06.151689    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-cac82f/auth-test-model-cac82f-kserve-7b4cbcc647-54ghx"]
Jul 14 12:21:06.162757 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:21:06.162722    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-cac82f/auth-test-model-cac82f-kserve-7b4cbcc647-54ghx"]
Jul 14 12:21:06.613487 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:21:06.613444    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-480a7f/auth-test-model-480a7f-kserve-54b4749f47-njtpv" podUID="863b6c1f-91bc-4103-8d1a-d12b5f265730" containerName="main" containerID="cri-o://2091a798edd236af61021521e91352f225ac7c850cb8ea06fe1376c1c0615342" gracePeriod=45
Jul 14 12:21:06.878926 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:21:06.878898    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-480a7f/auth-test-model-480a7f-kserve-54b4749f47-njtpv"
Jul 14 12:21:06.988088 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:21:06.988051    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/863b6c1f-91bc-4103-8d1a-d12b5f265730-model-cache\") pod \"863b6c1f-91bc-4103-8d1a-d12b5f265730\" (UID: \"863b6c1f-91bc-4103-8d1a-d12b5f265730\") "
Jul 14 12:21:06.988302 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:21:06.988143    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8gvbk\" (UniqueName: \"kubernetes.io/projected/863b6c1f-91bc-4103-8d1a-d12b5f265730-kube-api-access-8gvbk\") pod \"863b6c1f-91bc-4103-8d1a-d12b5f265730\" (UID: \"863b6c1f-91bc-4103-8d1a-d12b5f265730\") "
Jul 14 12:21:06.988302 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:21:06.988164    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/863b6c1f-91bc-4103-8d1a-d12b5f265730-tls-certs\") pod \"863b6c1f-91bc-4103-8d1a-d12b5f265730\" (UID: \"863b6c1f-91bc-4103-8d1a-d12b5f265730\") "
Jul 14 12:21:06.988302 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:21:06.988179    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/863b6c1f-91bc-4103-8d1a-d12b5f265730-home\") pod \"863b6c1f-91bc-4103-8d1a-d12b5f265730\" (UID: \"863b6c1f-91bc-4103-8d1a-d12b5f265730\") "
Jul 14 12:21:06.988302 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:21:06.988219    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/863b6c1f-91bc-4103-8d1a-d12b5f265730-dshm\") pod \"863b6c1f-91bc-4103-8d1a-d12b5f265730\" (UID: \"863b6c1f-91bc-4103-8d1a-d12b5f265730\") "
Jul 14 12:21:06.988302 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:21:06.988241    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/863b6c1f-91bc-4103-8d1a-d12b5f265730-tmp-dir\") pod \"863b6c1f-91bc-4103-8d1a-d12b5f265730\" (UID: \"863b6c1f-91bc-4103-8d1a-d12b5f265730\") "
Jul 14 12:21:06.988573 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:21:06.988360    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/863b6c1f-91bc-4103-8d1a-d12b5f265730-model-cache" (OuterVolumeSpecName: "model-cache") pod "863b6c1f-91bc-4103-8d1a-d12b5f265730" (UID: "863b6c1f-91bc-4103-8d1a-d12b5f265730"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 12:21:06.988573 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:21:06.988461    2551 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/863b6c1f-91bc-4103-8d1a-d12b5f265730-model-cache\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 12:21:06.988573 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:21:06.988464    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/863b6c1f-91bc-4103-8d1a-d12b5f265730-home" (OuterVolumeSpecName: "home") pod "863b6c1f-91bc-4103-8d1a-d12b5f265730" (UID: "863b6c1f-91bc-4103-8d1a-d12b5f265730"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 12:21:06.988741 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:21:06.988607    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/863b6c1f-91bc-4103-8d1a-d12b5f265730-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "863b6c1f-91bc-4103-8d1a-d12b5f265730" (UID: "863b6c1f-91bc-4103-8d1a-d12b5f265730"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 12:21:06.990564 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:21:06.990541    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/863b6c1f-91bc-4103-8d1a-d12b5f265730-dshm" (OuterVolumeSpecName: "dshm") pod "863b6c1f-91bc-4103-8d1a-d12b5f265730" (UID: "863b6c1f-91bc-4103-8d1a-d12b5f265730"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 12:21:06.990564 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:21:06.990552    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/863b6c1f-91bc-4103-8d1a-d12b5f265730-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "863b6c1f-91bc-4103-8d1a-d12b5f265730" (UID: "863b6c1f-91bc-4103-8d1a-d12b5f265730"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 12:21:06.990705 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:21:06.990617    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/863b6c1f-91bc-4103-8d1a-d12b5f265730-kube-api-access-8gvbk" (OuterVolumeSpecName: "kube-api-access-8gvbk") pod "863b6c1f-91bc-4103-8d1a-d12b5f265730" (UID: "863b6c1f-91bc-4103-8d1a-d12b5f265730"). InnerVolumeSpecName "kube-api-access-8gvbk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 12:21:07.089609 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:21:07.089531    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gvbk\" (UniqueName: \"kubernetes.io/projected/863b6c1f-91bc-4103-8d1a-d12b5f265730-kube-api-access-8gvbk\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 12:21:07.089609 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:21:07.089574    2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/863b6c1f-91bc-4103-8d1a-d12b5f265730-tls-certs\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 12:21:07.089609 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:21:07.089624    2551 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/863b6c1f-91bc-4103-8d1a-d12b5f265730-home\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 12:21:07.089609 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:21:07.089631    2551 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/863b6c1f-91bc-4103-8d1a-d12b5f265730-dshm\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 12:21:07.089907 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:21:07.089639    2551 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/863b6c1f-91bc-4103-8d1a-d12b5f265730-tmp-dir\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 12:21:07.117970 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:21:07.117868    2551 generic.go:356] "Generic (PLEG): container finished" podID="863b6c1f-91bc-4103-8d1a-d12b5f265730" containerID="2091a798edd236af61021521e91352f225ac7c850cb8ea06fe1376c1c0615342" exitCode=0
Jul 14 12:21:07.117970 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:21:07.117920    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-480a7f/auth-test-model-480a7f-kserve-54b4749f47-njtpv"
Jul 14 12:21:07.117970 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:21:07.117952    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-480a7f/auth-test-model-480a7f-kserve-54b4749f47-njtpv" event={"ID":"863b6c1f-91bc-4103-8d1a-d12b5f265730","Type":"ContainerDied","Data":"2091a798edd236af61021521e91352f225ac7c850cb8ea06fe1376c1c0615342"}
Jul 14 12:21:07.118262 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:21:07.117995    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-480a7f/auth-test-model-480a7f-kserve-54b4749f47-njtpv" event={"ID":"863b6c1f-91bc-4103-8d1a-d12b5f265730","Type":"ContainerDied","Data":"15f50e55136eed1fbab8770dee084b1326055390990587900ff3e8192654eb27"}
Jul 14 12:21:07.118262 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:21:07.118015    2551 scope.go:117] "RemoveContainer" containerID="2091a798edd236af61021521e91352f225ac7c850cb8ea06fe1376c1c0615342"
Jul 14 12:21:07.127685 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:21:07.127662    2551 scope.go:117] "RemoveContainer" containerID="2091a798edd236af61021521e91352f225ac7c850cb8ea06fe1376c1c0615342"
Jul 14 12:21:07.127993 ip-10-0-136-173 kubenswrapper[2551]: E0714 12:21:07.127973    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2091a798edd236af61021521e91352f225ac7c850cb8ea06fe1376c1c0615342\": container with ID starting with 2091a798edd236af61021521e91352f225ac7c850cb8ea06fe1376c1c0615342 not found: ID does not exist" containerID="2091a798edd236af61021521e91352f225ac7c850cb8ea06fe1376c1c0615342"
Jul 14 12:21:07.128066 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:21:07.128006    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2091a798edd236af61021521e91352f225ac7c850cb8ea06fe1376c1c0615342"} err="failed to get container status \"2091a798edd236af61021521e91352f225ac7c850cb8ea06fe1376c1c0615342\": rpc error: code = NotFound desc = could not find container \"2091a798edd236af61021521e91352f225ac7c850cb8ea06fe1376c1c0615342\": container with ID starting with 2091a798edd236af61021521e91352f225ac7c850cb8ea06fe1376c1c0615342 not found: ID does not exist"
Jul 14 12:21:07.151632 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:21:07.151573    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-480a7f/auth-test-model-480a7f-kserve-54b4749f47-njtpv"]
Jul 14 12:21:07.160540 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:21:07.160505    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-480a7f/auth-test-model-480a7f-kserve-54b4749f47-njtpv"]
Jul 14 12:21:07.191519 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:21:07.191488    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="863b6c1f-91bc-4103-8d1a-d12b5f265730" path="/var/lib/kubelet/pods/863b6c1f-91bc-4103-8d1a-d12b5f265730/volumes"
Jul 14 12:21:07.191902 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:21:07.191887    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8855a412-12cd-4c5b-a542-c7374fcbec62" path="/var/lib/kubelet/pods/8855a412-12cd-4c5b-a542-c7374fcbec62/volumes"
Jul 14 12:21:12.905196 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:21:12.905157    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 12:21:14.724550 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:21:14.724515    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:21:14.826396 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:21:14.826359    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:21:23.203624 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:21:23.203563    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-480a7f/sub-test-model-480a7f-kserve-7f99f87bc-x5t2j"]
Jul 14 12:21:23.204072 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:21:23.203917    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8855a412-12cd-4c5b-a542-c7374fcbec62" containerName="main"
Jul 14 12:21:23.204072 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:21:23.203929    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="8855a412-12cd-4c5b-a542-c7374fcbec62" containerName="main"
Jul 14 12:21:23.204072 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:21:23.203952    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="863b6c1f-91bc-4103-8d1a-d12b5f265730" containerName="main"
Jul 14 12:21:23.204072 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:21:23.203958    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="863b6c1f-91bc-4103-8d1a-d12b5f265730" containerName="main"
Jul 14 12:21:23.204072 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:21:23.204011    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="863b6c1f-91bc-4103-8d1a-d12b5f265730" containerName="main"
Jul 14 12:21:23.204072 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:21:23.204018    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="8855a412-12cd-4c5b-a542-c7374fcbec62" containerName="main"
Jul 14 12:21:23.207116 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:21:23.207080    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-480a7f/sub-test-model-480a7f-kserve-7f99f87bc-x5t2j"
Jul 14 12:21:23.209749 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:21:23.209721    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-480a7f\"/\"sub-test-model-480a7f-kserve-self-signed-certs\""
Jul 14 12:21:23.210970 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:21:23.210951    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-480a7f\"/\"kube-root-ca.crt\""
Jul 14 12:21:23.211081 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:21:23.211065    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-480a7f\"/\"openshift-service-ca.crt\""
Jul 14 12:21:23.211147 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:21:23.211124    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-480a7f\"/\"default-dockercfg-mc2m5\""
Jul 14 12:21:23.216452 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:21:23.216429    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-480a7f/sub-test-model-480a7f-kserve-7f99f87bc-x5t2j"]
Jul 14 12:21:23.218303 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:21:23.218280    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffd5eb0d-2f94-46c1-8809-5b4cc4bfe2d2-home\") pod \"sub-test-model-480a7f-kserve-7f99f87bc-x5t2j\" (UID: \"ffd5eb0d-2f94-46c1-8809-5b4cc4bfe2d2\") " pod="ai-tenant-e2e-shared-b-480a7f/sub-test-model-480a7f-kserve-7f99f87bc-x5t2j"
Jul 14 12:21:23.218391 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:21:23.218314    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffd5eb0d-2f94-46c1-8809-5b4cc4bfe2d2-model-cache\") pod \"sub-test-model-480a7f-kserve-7f99f87bc-x5t2j\" (UID: \"ffd5eb0d-2f94-46c1-8809-5b4cc4bfe2d2\") " pod="ai-tenant-e2e-shared-b-480a7f/sub-test-model-480a7f-kserve-7f99f87bc-x5t2j"
Jul 14 12:21:23.218391 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:21:23.218333    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffd5eb0d-2f94-46c1-8809-5b4cc4bfe2d2-dshm\") pod \"sub-test-model-480a7f-kserve-7f99f87bc-x5t2j\" (UID: \"ffd5eb0d-2f94-46c1-8809-5b4cc4bfe2d2\") " pod="ai-tenant-e2e-shared-b-480a7f/sub-test-model-480a7f-kserve-7f99f87bc-x5t2j"
Jul 14 12:21:23.218391 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:21:23.218380    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpsvj\" (UniqueName: \"kubernetes.io/projected/ffd5eb0d-2f94-46c1-8809-5b4cc4bfe2d2-kube-api-access-wpsvj\") pod \"sub-test-model-480a7f-kserve-7f99f87bc-x5t2j\" (UID: \"ffd5eb0d-2f94-46c1-8809-5b4cc4bfe2d2\") " pod="ai-tenant-e2e-shared-b-480a7f/sub-test-model-480a7f-kserve-7f99f87bc-x5t2j"
Jul 14 12:21:23.218500 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:21:23.218428    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffd5eb0d-2f94-46c1-8809-5b4cc4bfe2d2-tls-certs\") pod \"sub-test-model-480a7f-kserve-7f99f87bc-x5t2j\" (UID: \"ffd5eb0d-2f94-46c1-8809-5b4cc4bfe2d2\") " pod="ai-tenant-e2e-shared-b-480a7f/sub-test-model-480a7f-kserve-7f99f87bc-x5t2j"
Jul 14 12:21:23.218558 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:21:23.218542    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffd5eb0d-2f94-46c1-8809-5b4cc4bfe2d2-tmp-dir\") pod \"sub-test-model-480a7f-kserve-7f99f87bc-x5t2j\" (UID: \"ffd5eb0d-2f94-46c1-8809-5b4cc4bfe2d2\") " pod="ai-tenant-e2e-shared-b-480a7f/sub-test-model-480a7f-kserve-7f99f87bc-x5t2j"
Jul 14 12:21:23.280281 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:21:23.280249    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:21:23.319856 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:21:23.319806    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffd5eb0d-2f94-46c1-8809-5b4cc4bfe2d2-tmp-dir\") pod \"sub-test-model-480a7f-kserve-7f99f87bc-x5t2j\" (UID: \"ffd5eb0d-2f94-46c1-8809-5b4cc4bfe2d2\") " pod="ai-tenant-e2e-shared-b-480a7f/sub-test-model-480a7f-kserve-7f99f87bc-x5t2j"
Jul 14 12:21:23.319856 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:21:23.319875    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffd5eb0d-2f94-46c1-8809-5b4cc4bfe2d2-home\") pod \"sub-test-model-480a7f-kserve-7f99f87bc-x5t2j\" (UID: \"ffd5eb0d-2f94-46c1-8809-5b4cc4bfe2d2\") " pod="ai-tenant-e2e-shared-b-480a7f/sub-test-model-480a7f-kserve-7f99f87bc-x5t2j"
Jul 14 12:21:23.320136 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:21:23.319896    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffd5eb0d-2f94-46c1-8809-5b4cc4bfe2d2-model-cache\") pod \"sub-test-model-480a7f-kserve-7f99f87bc-x5t2j\" (UID: \"ffd5eb0d-2f94-46c1-8809-5b4cc4bfe2d2\") " pod="ai-tenant-e2e-shared-b-480a7f/sub-test-model-480a7f-kserve-7f99f87bc-x5t2j"
Jul 14 12:21:23.320136 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:21:23.319915    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffd5eb0d-2f94-46c1-8809-5b4cc4bfe2d2-dshm\") pod \"sub-test-model-480a7f-kserve-7f99f87bc-x5t2j\" (UID: \"ffd5eb0d-2f94-46c1-8809-5b4cc4bfe2d2\") " pod="ai-tenant-e2e-shared-b-480a7f/sub-test-model-480a7f-kserve-7f99f87bc-x5t2j"
Jul 14 12:21:23.320136 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:21:23.319932    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpsvj\" (UniqueName: \"kubernetes.io/projected/ffd5eb0d-2f94-46c1-8809-5b4cc4bfe2d2-kube-api-access-wpsvj\") pod \"sub-test-model-480a7f-kserve-7f99f87bc-x5t2j\" (UID: \"ffd5eb0d-2f94-46c1-8809-5b4cc4bfe2d2\") " pod="ai-tenant-e2e-shared-b-480a7f/sub-test-model-480a7f-kserve-7f99f87bc-x5t2j"
Jul 14 12:21:23.320136 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:21:23.319960    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffd5eb0d-2f94-46c1-8809-5b4cc4bfe2d2-tls-certs\") pod \"sub-test-model-480a7f-kserve-7f99f87bc-x5t2j\" (UID: \"ffd5eb0d-2f94-46c1-8809-5b4cc4bfe2d2\") " pod="ai-tenant-e2e-shared-b-480a7f/sub-test-model-480a7f-kserve-7f99f87bc-x5t2j"
Jul 14 12:21:23.320343 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:21:23.320303    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffd5eb0d-2f94-46c1-8809-5b4cc4bfe2d2-tmp-dir\") pod \"sub-test-model-480a7f-kserve-7f99f87bc-x5t2j\" (UID: \"ffd5eb0d-2f94-46c1-8809-5b4cc4bfe2d2\") " pod="ai-tenant-e2e-shared-b-480a7f/sub-test-model-480a7f-kserve-7f99f87bc-x5t2j"
Jul 14 12:21:23.320389 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:21:23.320347    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffd5eb0d-2f94-46c1-8809-5b4cc4bfe2d2-home\") pod \"sub-test-model-480a7f-kserve-7f99f87bc-x5t2j\" (UID: \"ffd5eb0d-2f94-46c1-8809-5b4cc4bfe2d2\") " pod="ai-tenant-e2e-shared-b-480a7f/sub-test-model-480a7f-kserve-7f99f87bc-x5t2j"
Jul 14 12:21:23.320425 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:21:23.320395    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffd5eb0d-2f94-46c1-8809-5b4cc4bfe2d2-model-cache\") pod \"sub-test-model-480a7f-kserve-7f99f87bc-x5t2j\" (UID: \"ffd5eb0d-2f94-46c1-8809-5b4cc4bfe2d2\") " pod="ai-tenant-e2e-shared-b-480a7f/sub-test-model-480a7f-kserve-7f99f87bc-x5t2j"
Jul 14 12:21:23.322263 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:21:23.322237    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffd5eb0d-2f94-46c1-8809-5b4cc4bfe2d2-dshm\") pod \"sub-test-model-480a7f-kserve-7f99f87bc-x5t2j\" (UID: \"ffd5eb0d-2f94-46c1-8809-5b4cc4bfe2d2\") " pod="ai-tenant-e2e-shared-b-480a7f/sub-test-model-480a7f-kserve-7f99f87bc-x5t2j"
Jul 14 12:21:23.322415 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:21:23.322398    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffd5eb0d-2f94-46c1-8809-5b4cc4bfe2d2-tls-certs\") pod \"sub-test-model-480a7f-kserve-7f99f87bc-x5t2j\" (UID: \"ffd5eb0d-2f94-46c1-8809-5b4cc4bfe2d2\") " pod="ai-tenant-e2e-shared-b-480a7f/sub-test-model-480a7f-kserve-7f99f87bc-x5t2j"
Jul 14 12:21:23.331658 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:21:23.331619    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpsvj\" (UniqueName: \"kubernetes.io/projected/ffd5eb0d-2f94-46c1-8809-5b4cc4bfe2d2-kube-api-access-wpsvj\") pod \"sub-test-model-480a7f-kserve-7f99f87bc-x5t2j\" (UID: \"ffd5eb0d-2f94-46c1-8809-5b4cc4bfe2d2\") " pod="ai-tenant-e2e-shared-b-480a7f/sub-test-model-480a7f-kserve-7f99f87bc-x5t2j"
Jul 14 12:21:23.516620 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:21:23.516497    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-480a7f/sub-test-model-480a7f-kserve-7f99f87bc-x5t2j"
Jul 14 12:21:23.667797 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:21:23.667758    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-480a7f/sub-test-model-480a7f-kserve-7f99f87bc-x5t2j"]
Jul 14 12:21:23.671846 ip-10-0-136-173 kubenswrapper[2551]: W0714 12:21:23.671811    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podffd5eb0d_2f94_46c1_8809_5b4cc4bfe2d2.slice/crio-2502178f8cda63f9ba3d16fe8f1ec614f2a26f1e4a635deff260a2f3bdca6967 WatchSource:0}: Error finding container 2502178f8cda63f9ba3d16fe8f1ec614f2a26f1e4a635deff260a2f3bdca6967: Status 404 returned error can't find the container with id 2502178f8cda63f9ba3d16fe8f1ec614f2a26f1e4a635deff260a2f3bdca6967
Jul 14 12:21:24.209612 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:21:24.209542    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-480a7f/sub-test-model-480a7f-kserve-7f99f87bc-x5t2j" event={"ID":"ffd5eb0d-2f94-46c1-8809-5b4cc4bfe2d2","Type":"ContainerStarted","Data":"8a1cb4c5ef9a5b98e926268f3158006c8d04be0b98ca95800c7e85daf8e451f4"}
Jul 14 12:21:24.210025 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:21:24.209624    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-480a7f/sub-test-model-480a7f-kserve-7f99f87bc-x5t2j" event={"ID":"ffd5eb0d-2f94-46c1-8809-5b4cc4bfe2d2","Type":"ContainerStarted","Data":"2502178f8cda63f9ba3d16fe8f1ec614f2a26f1e4a635deff260a2f3bdca6967"}
Jul 14 12:21:24.230478 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:21:24.230424    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-480a7f/sub-test-model-480a7f-kserve-7f99f87bc-x5t2j" podStartSLOduration=1.2304053050000001 podStartE2EDuration="1.230405305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 12:21:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 12:21:24.230118792 +0000 UTC m=+2635.461928088" watchObservedRunningTime="2026-07-14 12:21:24.230405305 +0000 UTC m=+2635.462214602"
Jul 14 12:21:29.099406 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:21:29.099371    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:21:29.105267 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:21:29.105234    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 12:21:29.195819 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:21:29.195782    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:21:33.517555 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:21:33.517488    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-480a7f/sub-test-model-480a7f-kserve-7f99f87bc-x5t2j"
Jul 14 12:21:33.517555 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:21:33.517572    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-480a7f/sub-test-model-480a7f-kserve-7f99f87bc-x5t2j"
Jul 14 12:21:33.530660 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:21:33.530628    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-480a7f/sub-test-model-480a7f-kserve-7f99f87bc-x5t2j"
Jul 14 12:21:34.269326 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:21:34.269290    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-480a7f/sub-test-model-480a7f-kserve-7f99f87bc-x5t2j"
Jul 14 12:21:39.641306 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:21:39.641267    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:21:44.599011 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:21:44.598974    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 12:22:02.846685 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:02.846649    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:22:03.784951 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:03.784903    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-480a7f/sub-test-model-480a7f-kserve-7f99f87bc-x5t2j"]
Jul 14 12:22:03.805328 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:03.805299    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:22:06.612242 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:06.612213    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:22:07.443275 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:07.443231    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 12:22:17.078988 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:17.078951    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:22:18.804970 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:18.804932    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-480a7f/sub-test-model-480a7f-kserve-7f99f87bc-x5t2j" podUID="ffd5eb0d-2f94-46c1-8809-5b4cc4bfe2d2" containerName="main" containerID="cri-o://8a1cb4c5ef9a5b98e926268f3158006c8d04be0b98ca95800c7e85daf8e451f4" gracePeriod=45
Jul 14 12:22:19.067509 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:19.067440    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-480a7f/sub-test-model-480a7f-kserve-7f99f87bc-x5t2j"
Jul 14 12:22:19.121711 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:19.121671    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffd5eb0d-2f94-46c1-8809-5b4cc4bfe2d2-tmp-dir\") pod \"ffd5eb0d-2f94-46c1-8809-5b4cc4bfe2d2\" (UID: \"ffd5eb0d-2f94-46c1-8809-5b4cc4bfe2d2\") "
Jul 14 12:22:19.121711 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:19.121712    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffd5eb0d-2f94-46c1-8809-5b4cc4bfe2d2-home\") pod \"ffd5eb0d-2f94-46c1-8809-5b4cc4bfe2d2\" (UID: \"ffd5eb0d-2f94-46c1-8809-5b4cc4bfe2d2\") "
Jul 14 12:22:19.121980 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:19.121751    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffd5eb0d-2f94-46c1-8809-5b4cc4bfe2d2-model-cache\") pod \"ffd5eb0d-2f94-46c1-8809-5b4cc4bfe2d2\" (UID: \"ffd5eb0d-2f94-46c1-8809-5b4cc4bfe2d2\") "
Jul 14 12:22:19.121980 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:19.121779    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wpsvj\" (UniqueName: \"kubernetes.io/projected/ffd5eb0d-2f94-46c1-8809-5b4cc4bfe2d2-kube-api-access-wpsvj\") pod \"ffd5eb0d-2f94-46c1-8809-5b4cc4bfe2d2\" (UID: \"ffd5eb0d-2f94-46c1-8809-5b4cc4bfe2d2\") "
Jul 14 12:22:19.121980 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:19.121827    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffd5eb0d-2f94-46c1-8809-5b4cc4bfe2d2-tls-certs\") pod \"ffd5eb0d-2f94-46c1-8809-5b4cc4bfe2d2\" (UID: \"ffd5eb0d-2f94-46c1-8809-5b4cc4bfe2d2\") "
Jul 14 12:22:19.121980 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:19.121867    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffd5eb0d-2f94-46c1-8809-5b4cc4bfe2d2-dshm\") pod \"ffd5eb0d-2f94-46c1-8809-5b4cc4bfe2d2\" (UID: \"ffd5eb0d-2f94-46c1-8809-5b4cc4bfe2d2\") "
Jul 14 12:22:19.122205 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:19.122011    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffd5eb0d-2f94-46c1-8809-5b4cc4bfe2d2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ffd5eb0d-2f94-46c1-8809-5b4cc4bfe2d2" (UID: "ffd5eb0d-2f94-46c1-8809-5b4cc4bfe2d2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 12:22:19.122205 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:19.122060    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffd5eb0d-2f94-46c1-8809-5b4cc4bfe2d2-home" (OuterVolumeSpecName: "home") pod "ffd5eb0d-2f94-46c1-8809-5b4cc4bfe2d2" (UID: "ffd5eb0d-2f94-46c1-8809-5b4cc4bfe2d2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 12:22:19.122205 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:19.122070    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffd5eb0d-2f94-46c1-8809-5b4cc4bfe2d2-model-cache" (OuterVolumeSpecName: "model-cache") pod "ffd5eb0d-2f94-46c1-8809-5b4cc4bfe2d2" (UID: "ffd5eb0d-2f94-46c1-8809-5b4cc4bfe2d2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 12:22:19.122205 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:19.122140    2551 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffd5eb0d-2f94-46c1-8809-5b4cc4bfe2d2-model-cache\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 12:22:19.122205 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:19.122158    2551 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffd5eb0d-2f94-46c1-8809-5b4cc4bfe2d2-tmp-dir\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 12:22:19.122205 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:19.122167    2551 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffd5eb0d-2f94-46c1-8809-5b4cc4bfe2d2-home\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 12:22:19.124113 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:19.124086    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffd5eb0d-2f94-46c1-8809-5b4cc4bfe2d2-dshm" (OuterVolumeSpecName: "dshm") pod "ffd5eb0d-2f94-46c1-8809-5b4cc4bfe2d2" (UID: "ffd5eb0d-2f94-46c1-8809-5b4cc4bfe2d2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 12:22:19.124753 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:19.124719    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffd5eb0d-2f94-46c1-8809-5b4cc4bfe2d2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ffd5eb0d-2f94-46c1-8809-5b4cc4bfe2d2" (UID: "ffd5eb0d-2f94-46c1-8809-5b4cc4bfe2d2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 12:22:19.124889 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:19.124795    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffd5eb0d-2f94-46c1-8809-5b4cc4bfe2d2-kube-api-access-wpsvj" (OuterVolumeSpecName: "kube-api-access-wpsvj") pod "ffd5eb0d-2f94-46c1-8809-5b4cc4bfe2d2" (UID: "ffd5eb0d-2f94-46c1-8809-5b4cc4bfe2d2"). InnerVolumeSpecName "kube-api-access-wpsvj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 12:22:19.223437 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:19.223399    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpsvj\" (UniqueName: \"kubernetes.io/projected/ffd5eb0d-2f94-46c1-8809-5b4cc4bfe2d2-kube-api-access-wpsvj\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 12:22:19.223437 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:19.223434    2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffd5eb0d-2f94-46c1-8809-5b4cc4bfe2d2-tls-certs\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 12:22:19.223437 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:19.223445    2551 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffd5eb0d-2f94-46c1-8809-5b4cc4bfe2d2-dshm\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 12:22:19.475817 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:19.475776    2551 generic.go:356] "Generic (PLEG): container finished" podID="ffd5eb0d-2f94-46c1-8809-5b4cc4bfe2d2" containerID="8a1cb4c5ef9a5b98e926268f3158006c8d04be0b98ca95800c7e85daf8e451f4" exitCode=0
Jul 14 12:22:19.476035 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:19.475840    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-480a7f/sub-test-model-480a7f-kserve-7f99f87bc-x5t2j"
Jul 14 12:22:19.476035 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:19.475859    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-480a7f/sub-test-model-480a7f-kserve-7f99f87bc-x5t2j" event={"ID":"ffd5eb0d-2f94-46c1-8809-5b4cc4bfe2d2","Type":"ContainerDied","Data":"8a1cb4c5ef9a5b98e926268f3158006c8d04be0b98ca95800c7e85daf8e451f4"}
Jul 14 12:22:19.476035 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:19.475918    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-480a7f/sub-test-model-480a7f-kserve-7f99f87bc-x5t2j" event={"ID":"ffd5eb0d-2f94-46c1-8809-5b4cc4bfe2d2","Type":"ContainerDied","Data":"2502178f8cda63f9ba3d16fe8f1ec614f2a26f1e4a635deff260a2f3bdca6967"}
Jul 14 12:22:19.476035 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:19.475941    2551 scope.go:117] "RemoveContainer" containerID="8a1cb4c5ef9a5b98e926268f3158006c8d04be0b98ca95800c7e85daf8e451f4"
Jul 14 12:22:19.484531 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:19.484508    2551 scope.go:117] "RemoveContainer" containerID="8a1cb4c5ef9a5b98e926268f3158006c8d04be0b98ca95800c7e85daf8e451f4"
Jul 14 12:22:19.484818 ip-10-0-136-173 kubenswrapper[2551]: E0714 12:22:19.484796    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a1cb4c5ef9a5b98e926268f3158006c8d04be0b98ca95800c7e85daf8e451f4\": container with ID starting with 8a1cb4c5ef9a5b98e926268f3158006c8d04be0b98ca95800c7e85daf8e451f4 not found: ID does not exist" containerID="8a1cb4c5ef9a5b98e926268f3158006c8d04be0b98ca95800c7e85daf8e451f4"
Jul 14 12:22:19.484874 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:19.484826    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a1cb4c5ef9a5b98e926268f3158006c8d04be0b98ca95800c7e85daf8e451f4"} err="failed to get container status \"8a1cb4c5ef9a5b98e926268f3158006c8d04be0b98ca95800c7e85daf8e451f4\": rpc error: code = NotFound desc = could not find container \"8a1cb4c5ef9a5b98e926268f3158006c8d04be0b98ca95800c7e85daf8e451f4\": container with ID starting with 8a1cb4c5ef9a5b98e926268f3158006c8d04be0b98ca95800c7e85daf8e451f4 not found: ID does not exist"
Jul 14 12:22:19.500549 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:19.500516    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-480a7f/sub-test-model-480a7f-kserve-7f99f87bc-x5t2j"]
Jul 14 12:22:19.508349 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:19.508320    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-480a7f/sub-test-model-480a7f-kserve-7f99f87bc-x5t2j"]
Jul 14 12:22:21.191374 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:21.191290    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ffd5eb0d-2f94-46c1-8809-5b4cc4bfe2d2" path="/var/lib/kubelet/pods/ffd5eb0d-2f94-46c1-8809-5b4cc4bfe2d2/volumes"
Jul 14 12:22:23.084861 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:23.084828    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-cac82f/rate-test-model-cac82f-kserve-594d78d9d7-lvr7m"]
Jul 14 12:22:23.085275 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:23.085187    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffd5eb0d-2f94-46c1-8809-5b4cc4bfe2d2" containerName="main"
Jul 14 12:22:23.085275 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:23.085198    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffd5eb0d-2f94-46c1-8809-5b4cc4bfe2d2" containerName="main"
Jul 14 12:22:23.085275 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:23.085261    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="ffd5eb0d-2f94-46c1-8809-5b4cc4bfe2d2" containerName="main"
Jul 14 12:22:23.089764 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:23.089743    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-cac82f/rate-test-model-cac82f-kserve-594d78d9d7-lvr7m"
Jul 14 12:22:23.092550 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:23.092526    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-cac82f\"/\"kube-root-ca.crt\""
Jul 14 12:22:23.092681 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:23.092608    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-cac82f\"/\"openshift-service-ca.crt\""
Jul 14 12:22:23.093958 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:23.093935    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-cac82f\"/\"rate-test-model-cac82f-kserve-self-signed-certs\""
Jul 14 12:22:23.093958 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:23.093948    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-cac82f\"/\"default-dockercfg-px7hg\""
Jul 14 12:22:23.102692 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:23.102665    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-cac82f/rate-test-model-cac82f-kserve-594d78d9d7-lvr7m"]
Jul 14 12:22:23.116006 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:23.115978    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:22:23.160393 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:23.160360    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/969427af-f358-4a9c-b2fb-dbdd7b80f7df-home\") pod \"rate-test-model-cac82f-kserve-594d78d9d7-lvr7m\" (UID: \"969427af-f358-4a9c-b2fb-dbdd7b80f7df\") " pod="ai-tenant-e2e-shared-a-cac82f/rate-test-model-cac82f-kserve-594d78d9d7-lvr7m"
Jul 14 12:22:23.160654 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:23.160407    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/969427af-f358-4a9c-b2fb-dbdd7b80f7df-model-cache\") pod \"rate-test-model-cac82f-kserve-594d78d9d7-lvr7m\" (UID: \"969427af-f358-4a9c-b2fb-dbdd7b80f7df\") " pod="ai-tenant-e2e-shared-a-cac82f/rate-test-model-cac82f-kserve-594d78d9d7-lvr7m"
Jul 14 12:22:23.160654 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:23.160429    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/969427af-f358-4a9c-b2fb-dbdd7b80f7df-tmp-dir\") pod \"rate-test-model-cac82f-kserve-594d78d9d7-lvr7m\" (UID: \"969427af-f358-4a9c-b2fb-dbdd7b80f7df\") " pod="ai-tenant-e2e-shared-a-cac82f/rate-test-model-cac82f-kserve-594d78d9d7-lvr7m"
Jul 14 12:22:23.160654 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:23.160451    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8lct\" (UniqueName: \"kubernetes.io/projected/969427af-f358-4a9c-b2fb-dbdd7b80f7df-kube-api-access-c8lct\") pod \"rate-test-model-cac82f-kserve-594d78d9d7-lvr7m\" (UID: \"969427af-f358-4a9c-b2fb-dbdd7b80f7df\") " pod="ai-tenant-e2e-shared-a-cac82f/rate-test-model-cac82f-kserve-594d78d9d7-lvr7m"
Jul 14 12:22:23.160654 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:23.160603    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/969427af-f358-4a9c-b2fb-dbdd7b80f7df-tls-certs\") pod \"rate-test-model-cac82f-kserve-594d78d9d7-lvr7m\" (UID: \"969427af-f358-4a9c-b2fb-dbdd7b80f7df\") " pod="ai-tenant-e2e-shared-a-cac82f/rate-test-model-cac82f-kserve-594d78d9d7-lvr7m"
Jul 14 12:22:23.160654 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:23.160649    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/969427af-f358-4a9c-b2fb-dbdd7b80f7df-dshm\") pod \"rate-test-model-cac82f-kserve-594d78d9d7-lvr7m\" (UID: \"969427af-f358-4a9c-b2fb-dbdd7b80f7df\") " pod="ai-tenant-e2e-shared-a-cac82f/rate-test-model-cac82f-kserve-594d78d9d7-lvr7m"
Jul 14 12:22:23.261422 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:23.261383    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/969427af-f358-4a9c-b2fb-dbdd7b80f7df-home\") pod \"rate-test-model-cac82f-kserve-594d78d9d7-lvr7m\" (UID: \"969427af-f358-4a9c-b2fb-dbdd7b80f7df\") " pod="ai-tenant-e2e-shared-a-cac82f/rate-test-model-cac82f-kserve-594d78d9d7-lvr7m"
Jul 14 12:22:23.261638 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:23.261434    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/969427af-f358-4a9c-b2fb-dbdd7b80f7df-model-cache\") pod \"rate-test-model-cac82f-kserve-594d78d9d7-lvr7m\" (UID: \"969427af-f358-4a9c-b2fb-dbdd7b80f7df\") " pod="ai-tenant-e2e-shared-a-cac82f/rate-test-model-cac82f-kserve-594d78d9d7-lvr7m"
Jul 14 12:22:23.261638 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:23.261451    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/969427af-f358-4a9c-b2fb-dbdd7b80f7df-tmp-dir\") pod \"rate-test-model-cac82f-kserve-594d78d9d7-lvr7m\" (UID: \"969427af-f358-4a9c-b2fb-dbdd7b80f7df\") " pod="ai-tenant-e2e-shared-a-cac82f/rate-test-model-cac82f-kserve-594d78d9d7-lvr7m"
Jul 14 12:22:23.261638 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:23.261605    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8lct\" (UniqueName: \"kubernetes.io/projected/969427af-f358-4a9c-b2fb-dbdd7b80f7df-kube-api-access-c8lct\") pod \"rate-test-model-cac82f-kserve-594d78d9d7-lvr7m\" (UID: \"969427af-f358-4a9c-b2fb-dbdd7b80f7df\") " pod="ai-tenant-e2e-shared-a-cac82f/rate-test-model-cac82f-kserve-594d78d9d7-lvr7m"
Jul 14 12:22:23.261898 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:23.261758    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/969427af-f358-4a9c-b2fb-dbdd7b80f7df-tls-certs\") pod \"rate-test-model-cac82f-kserve-594d78d9d7-lvr7m\" (UID: \"969427af-f358-4a9c-b2fb-dbdd7b80f7df\") " pod="ai-tenant-e2e-shared-a-cac82f/rate-test-model-cac82f-kserve-594d78d9d7-lvr7m"
Jul 14 12:22:23.261898 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:23.261810    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/969427af-f358-4a9c-b2fb-dbdd7b80f7df-dshm\") pod \"rate-test-model-cac82f-kserve-594d78d9d7-lvr7m\" (UID: \"969427af-f358-4a9c-b2fb-dbdd7b80f7df\") " pod="ai-tenant-e2e-shared-a-cac82f/rate-test-model-cac82f-kserve-594d78d9d7-lvr7m"
Jul 14 12:22:23.261898 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:23.261854    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/969427af-f358-4a9c-b2fb-dbdd7b80f7df-tmp-dir\") pod \"rate-test-model-cac82f-kserve-594d78d9d7-lvr7m\" (UID: \"969427af-f358-4a9c-b2fb-dbdd7b80f7df\") " pod="ai-tenant-e2e-shared-a-cac82f/rate-test-model-cac82f-kserve-594d78d9d7-lvr7m"
Jul 14 12:22:23.262067 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:23.261909    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/969427af-f358-4a9c-b2fb-dbdd7b80f7df-home\") pod \"rate-test-model-cac82f-kserve-594d78d9d7-lvr7m\" (UID: \"969427af-f358-4a9c-b2fb-dbdd7b80f7df\") " pod="ai-tenant-e2e-shared-a-cac82f/rate-test-model-cac82f-kserve-594d78d9d7-lvr7m"
Jul 14 12:22:23.262067 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:23.261951    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/969427af-f358-4a9c-b2fb-dbdd7b80f7df-model-cache\") pod \"rate-test-model-cac82f-kserve-594d78d9d7-lvr7m\" (UID: \"969427af-f358-4a9c-b2fb-dbdd7b80f7df\") " pod="ai-tenant-e2e-shared-a-cac82f/rate-test-model-cac82f-kserve-594d78d9d7-lvr7m"
Jul 14 12:22:23.264077 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:23.264055    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/969427af-f358-4a9c-b2fb-dbdd7b80f7df-dshm\") pod \"rate-test-model-cac82f-kserve-594d78d9d7-lvr7m\" (UID: \"969427af-f358-4a9c-b2fb-dbdd7b80f7df\") " pod="ai-tenant-e2e-shared-a-cac82f/rate-test-model-cac82f-kserve-594d78d9d7-lvr7m"
Jul 14 12:22:23.264384 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:23.264362    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/969427af-f358-4a9c-b2fb-dbdd7b80f7df-tls-certs\") pod \"rate-test-model-cac82f-kserve-594d78d9d7-lvr7m\" (UID: \"969427af-f358-4a9c-b2fb-dbdd7b80f7df\") " pod="ai-tenant-e2e-shared-a-cac82f/rate-test-model-cac82f-kserve-594d78d9d7-lvr7m"
Jul 14 12:22:23.272976 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:23.272952    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8lct\" (UniqueName: \"kubernetes.io/projected/969427af-f358-4a9c-b2fb-dbdd7b80f7df-kube-api-access-c8lct\") pod \"rate-test-model-cac82f-kserve-594d78d9d7-lvr7m\" (UID: \"969427af-f358-4a9c-b2fb-dbdd7b80f7df\") " pod="ai-tenant-e2e-shared-a-cac82f/rate-test-model-cac82f-kserve-594d78d9d7-lvr7m"
Jul 14 12:22:23.400250 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:23.400142    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-cac82f/rate-test-model-cac82f-kserve-594d78d9d7-lvr7m"
Jul 14 12:22:23.568791 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:23.568756    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-cac82f/rate-test-model-cac82f-kserve-594d78d9d7-lvr7m"]
Jul 14 12:22:23.571810 ip-10-0-136-173 kubenswrapper[2551]: W0714 12:22:23.571775    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod969427af_f358_4a9c_b2fb_dbdd7b80f7df.slice/crio-806b5b6cea68f0b3c24fc06b07171c5f95157c34cfc6942086b664104bcab670 WatchSource:0}: Error finding container 806b5b6cea68f0b3c24fc06b07171c5f95157c34cfc6942086b664104bcab670: Status 404 returned error can't find the container with id 806b5b6cea68f0b3c24fc06b07171c5f95157c34cfc6942086b664104bcab670
Jul 14 12:22:24.502237 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:24.502195    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-cac82f/rate-test-model-cac82f-kserve-594d78d9d7-lvr7m" event={"ID":"969427af-f358-4a9c-b2fb-dbdd7b80f7df","Type":"ContainerStarted","Data":"1edcc38596e7bfc6c24f1427b644eb1d4bf644d1205f65c18302436181f771f6"}
Jul 14 12:22:24.502237 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:24.502244    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-cac82f/rate-test-model-cac82f-kserve-594d78d9d7-lvr7m" event={"ID":"969427af-f358-4a9c-b2fb-dbdd7b80f7df","Type":"ContainerStarted","Data":"806b5b6cea68f0b3c24fc06b07171c5f95157c34cfc6942086b664104bcab670"}
Jul 14 12:22:24.526933 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:24.526873    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-cac82f/rate-test-model-cac82f-kserve-594d78d9d7-lvr7m" podStartSLOduration=1.52685399 podStartE2EDuration="1.52685399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 12:22:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 12:22:24.524656576 +0000 UTC m=+2695.756465913" watchObservedRunningTime="2026-07-14 12:22:24.52685399 +0000 UTC m=+2695.758663286"
Jul 14 12:22:27.399813 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:27.399772    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 12:22:29.278148 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:29.278123    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cp57b_62c98a11-ece4-4497-99e7-0840a7626a56/ovn-acl-logging/0.log"
Jul 14 12:22:29.287982 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:29.287955    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cp57b_62c98a11-ece4-4497-99e7-0840a7626a56/ovn-acl-logging/0.log"
Jul 14 12:22:33.400565 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:33.400517    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-cac82f/rate-test-model-cac82f-kserve-594d78d9d7-lvr7m"
Jul 14 12:22:33.401085 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:33.400601    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-cac82f/rate-test-model-cac82f-kserve-594d78d9d7-lvr7m"
Jul 14 12:22:33.413359 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:33.413329    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-cac82f/rate-test-model-cac82f-kserve-594d78d9d7-lvr7m"
Jul 14 12:22:33.558465 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:33.558433    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-cac82f/rate-test-model-cac82f-kserve-594d78d9d7-lvr7m"
Jul 14 12:22:44.211564 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:44.211528    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-480a7f/rate-test-model-480a7f-kserve-7cb6d59f47-76qqt"]
Jul 14 12:22:44.214828 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:44.214807    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-480a7f/rate-test-model-480a7f-kserve-7cb6d59f47-76qqt"
Jul 14 12:22:44.219571 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:44.219540    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-480a7f\"/\"rate-test-model-480a7f-kserve-self-signed-certs\""
Jul 14 12:22:44.219747 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:44.219626    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-480a7f\"/\"kube-root-ca.crt\""
Jul 14 12:22:44.219747 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:44.219649    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-480a7f\"/\"default-dockercfg-mc2m5\""
Jul 14 12:22:44.219747 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:44.219723    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-480a7f\"/\"openshift-service-ca.crt\""
Jul 14 12:22:44.226480 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:44.226451    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-480a7f/rate-test-model-480a7f-kserve-7cb6d59f47-76qqt"]
Jul 14 12:22:44.310244 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:44.310198    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:22:44.358537 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:44.358485    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ldbh\" (UniqueName: \"kubernetes.io/projected/cd6d46ec-da0d-450a-b83e-d25f47f9add2-kube-api-access-8ldbh\") pod \"rate-test-model-480a7f-kserve-7cb6d59f47-76qqt\" (UID: \"cd6d46ec-da0d-450a-b83e-d25f47f9add2\") " pod="ai-tenant-e2e-shared-b-480a7f/rate-test-model-480a7f-kserve-7cb6d59f47-76qqt"
Jul 14 12:22:44.358774 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:44.358624    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd6d46ec-da0d-450a-b83e-d25f47f9add2-dshm\") pod \"rate-test-model-480a7f-kserve-7cb6d59f47-76qqt\" (UID: \"cd6d46ec-da0d-450a-b83e-d25f47f9add2\") " pod="ai-tenant-e2e-shared-b-480a7f/rate-test-model-480a7f-kserve-7cb6d59f47-76qqt"
Jul 14 12:22:44.358774 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:44.358662    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd6d46ec-da0d-450a-b83e-d25f47f9add2-model-cache\") pod \"rate-test-model-480a7f-kserve-7cb6d59f47-76qqt\" (UID: \"cd6d46ec-da0d-450a-b83e-d25f47f9add2\") " pod="ai-tenant-e2e-shared-b-480a7f/rate-test-model-480a7f-kserve-7cb6d59f47-76qqt"
Jul 14 12:22:44.358774 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:44.358713    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd6d46ec-da0d-450a-b83e-d25f47f9add2-home\") pod \"rate-test-model-480a7f-kserve-7cb6d59f47-76qqt\" (UID: \"cd6d46ec-da0d-450a-b83e-d25f47f9add2\") " pod="ai-tenant-e2e-shared-b-480a7f/rate-test-model-480a7f-kserve-7cb6d59f47-76qqt"
Jul 14 12:22:44.358774 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:44.358733    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd6d46ec-da0d-450a-b83e-d25f47f9add2-tmp-dir\") pod \"rate-test-model-480a7f-kserve-7cb6d59f47-76qqt\" (UID: \"cd6d46ec-da0d-450a-b83e-d25f47f9add2\") " pod="ai-tenant-e2e-shared-b-480a7f/rate-test-model-480a7f-kserve-7cb6d59f47-76qqt"
Jul 14 12:22:44.359001 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:44.358811    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd6d46ec-da0d-450a-b83e-d25f47f9add2-tls-certs\") pod \"rate-test-model-480a7f-kserve-7cb6d59f47-76qqt\" (UID: \"cd6d46ec-da0d-450a-b83e-d25f47f9add2\") " pod="ai-tenant-e2e-shared-b-480a7f/rate-test-model-480a7f-kserve-7cb6d59f47-76qqt"
Jul 14 12:22:44.459784 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:44.459745    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd6d46ec-da0d-450a-b83e-d25f47f9add2-dshm\") pod \"rate-test-model-480a7f-kserve-7cb6d59f47-76qqt\" (UID: \"cd6d46ec-da0d-450a-b83e-d25f47f9add2\") " pod="ai-tenant-e2e-shared-b-480a7f/rate-test-model-480a7f-kserve-7cb6d59f47-76qqt"
Jul 14 12:22:44.459972 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:44.459791    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd6d46ec-da0d-450a-b83e-d25f47f9add2-model-cache\") pod \"rate-test-model-480a7f-kserve-7cb6d59f47-76qqt\" (UID: \"cd6d46ec-da0d-450a-b83e-d25f47f9add2\") " pod="ai-tenant-e2e-shared-b-480a7f/rate-test-model-480a7f-kserve-7cb6d59f47-76qqt"
Jul 14 12:22:44.459972 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:44.459852    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd6d46ec-da0d-450a-b83e-d25f47f9add2-home\") pod \"rate-test-model-480a7f-kserve-7cb6d59f47-76qqt\" (UID: \"cd6d46ec-da0d-450a-b83e-d25f47f9add2\") " pod="ai-tenant-e2e-shared-b-480a7f/rate-test-model-480a7f-kserve-7cb6d59f47-76qqt"
Jul 14 12:22:44.459972 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:44.459877    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd6d46ec-da0d-450a-b83e-d25f47f9add2-tmp-dir\") pod \"rate-test-model-480a7f-kserve-7cb6d59f47-76qqt\" (UID: \"cd6d46ec-da0d-450a-b83e-d25f47f9add2\") " pod="ai-tenant-e2e-shared-b-480a7f/rate-test-model-480a7f-kserve-7cb6d59f47-76qqt"
Jul 14 12:22:44.459972 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:44.459901    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd6d46ec-da0d-450a-b83e-d25f47f9add2-tls-certs\") pod \"rate-test-model-480a7f-kserve-7cb6d59f47-76qqt\" (UID: \"cd6d46ec-da0d-450a-b83e-d25f47f9add2\") " pod="ai-tenant-e2e-shared-b-480a7f/rate-test-model-480a7f-kserve-7cb6d59f47-76qqt"
Jul 14 12:22:44.459972 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:44.459955    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ldbh\" (UniqueName: \"kubernetes.io/projected/cd6d46ec-da0d-450a-b83e-d25f47f9add2-kube-api-access-8ldbh\") pod \"rate-test-model-480a7f-kserve-7cb6d59f47-76qqt\" (UID: \"cd6d46ec-da0d-450a-b83e-d25f47f9add2\") " pod="ai-tenant-e2e-shared-b-480a7f/rate-test-model-480a7f-kserve-7cb6d59f47-76qqt"
Jul 14 12:22:44.460361 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:44.460334    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd6d46ec-da0d-450a-b83e-d25f47f9add2-home\") pod \"rate-test-model-480a7f-kserve-7cb6d59f47-76qqt\" (UID: \"cd6d46ec-da0d-450a-b83e-d25f47f9add2\") " pod="ai-tenant-e2e-shared-b-480a7f/rate-test-model-480a7f-kserve-7cb6d59f47-76qqt"
Jul 14 12:22:44.460501 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:44.460369    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd6d46ec-da0d-450a-b83e-d25f47f9add2-tmp-dir\") pod \"rate-test-model-480a7f-kserve-7cb6d59f47-76qqt\" (UID: \"cd6d46ec-da0d-450a-b83e-d25f47f9add2\") " pod="ai-tenant-e2e-shared-b-480a7f/rate-test-model-480a7f-kserve-7cb6d59f47-76qqt"
Jul 14 12:22:44.460501 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:44.460341    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd6d46ec-da0d-450a-b83e-d25f47f9add2-model-cache\") pod \"rate-test-model-480a7f-kserve-7cb6d59f47-76qqt\" (UID: \"cd6d46ec-da0d-450a-b83e-d25f47f9add2\") " pod="ai-tenant-e2e-shared-b-480a7f/rate-test-model-480a7f-kserve-7cb6d59f47-76qqt"
Jul 14 12:22:44.462211 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:44.462156    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd6d46ec-da0d-450a-b83e-d25f47f9add2-dshm\") pod \"rate-test-model-480a7f-kserve-7cb6d59f47-76qqt\" (UID: \"cd6d46ec-da0d-450a-b83e-d25f47f9add2\") " pod="ai-tenant-e2e-shared-b-480a7f/rate-test-model-480a7f-kserve-7cb6d59f47-76qqt"
Jul 14 12:22:44.462749 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:44.462727    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd6d46ec-da0d-450a-b83e-d25f47f9add2-tls-certs\") pod \"rate-test-model-480a7f-kserve-7cb6d59f47-76qqt\" (UID: \"cd6d46ec-da0d-450a-b83e-d25f47f9add2\") " pod="ai-tenant-e2e-shared-b-480a7f/rate-test-model-480a7f-kserve-7cb6d59f47-76qqt"
Jul 14 12:22:44.473385 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:44.473359    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ldbh\" (UniqueName: \"kubernetes.io/projected/cd6d46ec-da0d-450a-b83e-d25f47f9add2-kube-api-access-8ldbh\") pod \"rate-test-model-480a7f-kserve-7cb6d59f47-76qqt\" (UID: \"cd6d46ec-da0d-450a-b83e-d25f47f9add2\") " pod="ai-tenant-e2e-shared-b-480a7f/rate-test-model-480a7f-kserve-7cb6d59f47-76qqt"
Jul 14 12:22:44.525011 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:44.524970    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-480a7f/rate-test-model-480a7f-kserve-7cb6d59f47-76qqt"
Jul 14 12:22:44.686724 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:44.686678    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-480a7f/rate-test-model-480a7f-kserve-7cb6d59f47-76qqt"]
Jul 14 12:22:44.689798 ip-10-0-136-173 kubenswrapper[2551]: W0714 12:22:44.689770    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcd6d46ec_da0d_450a_b83e_d25f47f9add2.slice/crio-b87486d5d395ef1e0f72422f02a8b7529085b0169a0d0dbc93aa588788a6b399 WatchSource:0}: Error finding container b87486d5d395ef1e0f72422f02a8b7529085b0169a0d0dbc93aa588788a6b399: Status 404 returned error can't find the container with id b87486d5d395ef1e0f72422f02a8b7529085b0169a0d0dbc93aa588788a6b399
Jul 14 12:22:45.607849 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:45.607812    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-480a7f/rate-test-model-480a7f-kserve-7cb6d59f47-76qqt" event={"ID":"cd6d46ec-da0d-450a-b83e-d25f47f9add2","Type":"ContainerStarted","Data":"deb1ab944e35dd37c516efe8bc8e9c115c2a7493803c3a009884ace77abb1c56"}
Jul 14 12:22:45.607849 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:45.607856    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-480a7f/rate-test-model-480a7f-kserve-7cb6d59f47-76qqt" event={"ID":"cd6d46ec-da0d-450a-b83e-d25f47f9add2","Type":"ContainerStarted","Data":"b87486d5d395ef1e0f72422f02a8b7529085b0169a0d0dbc93aa588788a6b399"}
Jul 14 12:22:45.633490 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:45.633418    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-480a7f/rate-test-model-480a7f-kserve-7cb6d59f47-76qqt" podStartSLOduration=1.6333994779999998 podStartE2EDuration="1.633399478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 12:22:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 12:22:45.632283863 +0000 UTC m=+2716.864093159" watchObservedRunningTime="2026-07-14 12:22:45.633399478 +0000 UTC m=+2716.865208774"
Jul 14 12:22:49.328085 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:49.328041    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 12:22:54.525396 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:54.525351    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-480a7f/rate-test-model-480a7f-kserve-7cb6d59f47-76qqt"
Jul 14 12:22:54.525951 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:54.525665    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-480a7f/rate-test-model-480a7f-kserve-7cb6d59f47-76qqt"
Jul 14 12:22:54.539877 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:54.539845    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-480a7f/rate-test-model-480a7f-kserve-7cb6d59f47-76qqt"
Jul 14 12:22:54.666405 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:22:54.666372    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-480a7f/rate-test-model-480a7f-kserve-7cb6d59f47-76qqt"
Jul 14 12:23:04.198250 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:04.198208    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 12:23:20.069201 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:20.069164    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:23:20.167158 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:20.167123    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:23:20.800063 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:20.800018    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 12:23:27.840691 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:27.840652    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-cac82f/rate-test-model-cac82f-kserve-594d78d9d7-lvr7m"]
Jul 14 12:23:27.872882 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:27.872845    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:23:27.982308 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:27.982267    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:23:28.117536 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:28.117449    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:23:28.689872 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:28.689835    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-480a7f/rate-test-model-480a7f-kserve-7cb6d59f47-76qqt"]
Jul 14 12:23:28.702288 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:28.702233    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:23:35.296437 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:35.296396    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 12:23:42.860651 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:42.860599    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-cac82f/rate-test-model-cac82f-kserve-594d78d9d7-lvr7m" podUID="969427af-f358-4a9c-b2fb-dbdd7b80f7df" containerName="main" containerID="cri-o://1edcc38596e7bfc6c24f1427b644eb1d4bf644d1205f65c18302436181f771f6" gracePeriod=45
Jul 14 12:23:43.123623 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:43.123519    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-cac82f/rate-test-model-cac82f-kserve-594d78d9d7-lvr7m"
Jul 14 12:23:43.307793 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:43.307753    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/969427af-f358-4a9c-b2fb-dbdd7b80f7df-home\") pod \"969427af-f358-4a9c-b2fb-dbdd7b80f7df\" (UID: \"969427af-f358-4a9c-b2fb-dbdd7b80f7df\") "
Jul 14 12:23:43.307991 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:43.307811    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/969427af-f358-4a9c-b2fb-dbdd7b80f7df-tmp-dir\") pod \"969427af-f358-4a9c-b2fb-dbdd7b80f7df\" (UID: \"969427af-f358-4a9c-b2fb-dbdd7b80f7df\") "
Jul 14 12:23:43.307991 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:43.307865    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c8lct\" (UniqueName: \"kubernetes.io/projected/969427af-f358-4a9c-b2fb-dbdd7b80f7df-kube-api-access-c8lct\") pod \"969427af-f358-4a9c-b2fb-dbdd7b80f7df\" (UID: \"969427af-f358-4a9c-b2fb-dbdd7b80f7df\") "
Jul 14 12:23:43.307991 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:43.307895    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/969427af-f358-4a9c-b2fb-dbdd7b80f7df-tls-certs\") pod \"969427af-f358-4a9c-b2fb-dbdd7b80f7df\" (UID: \"969427af-f358-4a9c-b2fb-dbdd7b80f7df\") "
Jul 14 12:23:43.307991 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:43.307922    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/969427af-f358-4a9c-b2fb-dbdd7b80f7df-model-cache\") pod \"969427af-f358-4a9c-b2fb-dbdd7b80f7df\" (UID: \"969427af-f358-4a9c-b2fb-dbdd7b80f7df\") "
Jul 14 12:23:43.307991 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:43.307989    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/969427af-f358-4a9c-b2fb-dbdd7b80f7df-dshm\") pod \"969427af-f358-4a9c-b2fb-dbdd7b80f7df\" (UID: \"969427af-f358-4a9c-b2fb-dbdd7b80f7df\") "
Jul 14 12:23:43.308254 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:43.308049    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/969427af-f358-4a9c-b2fb-dbdd7b80f7df-home" (OuterVolumeSpecName: "home") pod "969427af-f358-4a9c-b2fb-dbdd7b80f7df" (UID: "969427af-f358-4a9c-b2fb-dbdd7b80f7df"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 12:23:43.308254 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:43.308135    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/969427af-f358-4a9c-b2fb-dbdd7b80f7df-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "969427af-f358-4a9c-b2fb-dbdd7b80f7df" (UID: "969427af-f358-4a9c-b2fb-dbdd7b80f7df"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 12:23:43.308372 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:43.308267    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/969427af-f358-4a9c-b2fb-dbdd7b80f7df-model-cache" (OuterVolumeSpecName: "model-cache") pod "969427af-f358-4a9c-b2fb-dbdd7b80f7df" (UID: "969427af-f358-4a9c-b2fb-dbdd7b80f7df"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 12:23:43.308372 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:43.308322    2551 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/969427af-f358-4a9c-b2fb-dbdd7b80f7df-home\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 12:23:43.308372 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:43.308363    2551 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/969427af-f358-4a9c-b2fb-dbdd7b80f7df-tmp-dir\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 12:23:43.310148 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:43.310124    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/969427af-f358-4a9c-b2fb-dbdd7b80f7df-dshm" (OuterVolumeSpecName: "dshm") pod "969427af-f358-4a9c-b2fb-dbdd7b80f7df" (UID: "969427af-f358-4a9c-b2fb-dbdd7b80f7df"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 12:23:43.310468 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:43.310441    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/969427af-f358-4a9c-b2fb-dbdd7b80f7df-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "969427af-f358-4a9c-b2fb-dbdd7b80f7df" (UID: "969427af-f358-4a9c-b2fb-dbdd7b80f7df"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 12:23:43.310519 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:43.310453    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/969427af-f358-4a9c-b2fb-dbdd7b80f7df-kube-api-access-c8lct" (OuterVolumeSpecName: "kube-api-access-c8lct") pod "969427af-f358-4a9c-b2fb-dbdd7b80f7df" (UID: "969427af-f358-4a9c-b2fb-dbdd7b80f7df"). InnerVolumeSpecName "kube-api-access-c8lct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 12:23:43.409836 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:43.409719    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8lct\" (UniqueName: \"kubernetes.io/projected/969427af-f358-4a9c-b2fb-dbdd7b80f7df-kube-api-access-c8lct\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 12:23:43.409836 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:43.409773    2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/969427af-f358-4a9c-b2fb-dbdd7b80f7df-tls-certs\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 12:23:43.409836 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:43.409784    2551 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/969427af-f358-4a9c-b2fb-dbdd7b80f7df-model-cache\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 12:23:43.409836 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:43.409792    2551 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/969427af-f358-4a9c-b2fb-dbdd7b80f7df-dshm\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 12:23:43.710029 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:43.709957    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-480a7f/rate-test-model-480a7f-kserve-7cb6d59f47-76qqt" podUID="cd6d46ec-da0d-450a-b83e-d25f47f9add2" containerName="main" containerID="cri-o://deb1ab944e35dd37c516efe8bc8e9c115c2a7493803c3a009884ace77abb1c56" gracePeriod=45
Jul 14 12:23:43.899549 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:43.899512    2551 generic.go:356] "Generic (PLEG): container finished" podID="cd6d46ec-da0d-450a-b83e-d25f47f9add2" containerID="deb1ab944e35dd37c516efe8bc8e9c115c2a7493803c3a009884ace77abb1c56" exitCode=0
Jul 14 12:23:43.900015 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:43.899606    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-480a7f/rate-test-model-480a7f-kserve-7cb6d59f47-76qqt" event={"ID":"cd6d46ec-da0d-450a-b83e-d25f47f9add2","Type":"ContainerDied","Data":"deb1ab944e35dd37c516efe8bc8e9c115c2a7493803c3a009884ace77abb1c56"}
Jul 14 12:23:43.901304 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:43.901274    2551 generic.go:356] "Generic (PLEG): container finished" podID="969427af-f358-4a9c-b2fb-dbdd7b80f7df" containerID="1edcc38596e7bfc6c24f1427b644eb1d4bf644d1205f65c18302436181f771f6" exitCode=0
Jul 14 12:23:43.901451 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:43.901317    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-cac82f/rate-test-model-cac82f-kserve-594d78d9d7-lvr7m" event={"ID":"969427af-f358-4a9c-b2fb-dbdd7b80f7df","Type":"ContainerDied","Data":"1edcc38596e7bfc6c24f1427b644eb1d4bf644d1205f65c18302436181f771f6"}
Jul 14 12:23:43.901451 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:43.901348    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-cac82f/rate-test-model-cac82f-kserve-594d78d9d7-lvr7m" event={"ID":"969427af-f358-4a9c-b2fb-dbdd7b80f7df","Type":"ContainerDied","Data":"806b5b6cea68f0b3c24fc06b07171c5f95157c34cfc6942086b664104bcab670"}
Jul 14 12:23:43.901451 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:43.901369    2551 scope.go:117] "RemoveContainer" containerID="1edcc38596e7bfc6c24f1427b644eb1d4bf644d1205f65c18302436181f771f6"
Jul 14 12:23:43.901451 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:43.901369    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-cac82f/rate-test-model-cac82f-kserve-594d78d9d7-lvr7m"
Jul 14 12:23:43.914805 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:43.914780    2551 scope.go:117] "RemoveContainer" containerID="1edcc38596e7bfc6c24f1427b644eb1d4bf644d1205f65c18302436181f771f6"
Jul 14 12:23:43.915188 ip-10-0-136-173 kubenswrapper[2551]: E0714 12:23:43.915164    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1edcc38596e7bfc6c24f1427b644eb1d4bf644d1205f65c18302436181f771f6\": container with ID starting with 1edcc38596e7bfc6c24f1427b644eb1d4bf644d1205f65c18302436181f771f6 not found: ID does not exist" containerID="1edcc38596e7bfc6c24f1427b644eb1d4bf644d1205f65c18302436181f771f6"
Jul 14 12:23:43.915272 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:43.915201    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1edcc38596e7bfc6c24f1427b644eb1d4bf644d1205f65c18302436181f771f6"} err="failed to get container status \"1edcc38596e7bfc6c24f1427b644eb1d4bf644d1205f65c18302436181f771f6\": rpc error: code = NotFound desc = could not find container \"1edcc38596e7bfc6c24f1427b644eb1d4bf644d1205f65c18302436181f771f6\": container with ID starting with 1edcc38596e7bfc6c24f1427b644eb1d4bf644d1205f65c18302436181f771f6 not found: ID does not exist"
Jul 14 12:23:43.933876 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:43.933843    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-cac82f/rate-test-model-cac82f-kserve-594d78d9d7-lvr7m"]
Jul 14 12:23:43.936466 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:43.936433    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-cac82f/rate-test-model-cac82f-kserve-594d78d9d7-lvr7m"]
Jul 14 12:23:43.980752 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:43.980725    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-480a7f/rate-test-model-480a7f-kserve-7cb6d59f47-76qqt"
Jul 14 12:23:44.114466 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:44.114432    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8ldbh\" (UniqueName: \"kubernetes.io/projected/cd6d46ec-da0d-450a-b83e-d25f47f9add2-kube-api-access-8ldbh\") pod \"cd6d46ec-da0d-450a-b83e-d25f47f9add2\" (UID: \"cd6d46ec-da0d-450a-b83e-d25f47f9add2\") "
Jul 14 12:23:44.114466 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:44.114471    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd6d46ec-da0d-450a-b83e-d25f47f9add2-model-cache\") pod \"cd6d46ec-da0d-450a-b83e-d25f47f9add2\" (UID: \"cd6d46ec-da0d-450a-b83e-d25f47f9add2\") "
Jul 14 12:23:44.114774 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:44.114506    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd6d46ec-da0d-450a-b83e-d25f47f9add2-tls-certs\") pod \"cd6d46ec-da0d-450a-b83e-d25f47f9add2\" (UID: \"cd6d46ec-da0d-450a-b83e-d25f47f9add2\") "
Jul 14 12:23:44.114774 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:44.114562    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd6d46ec-da0d-450a-b83e-d25f47f9add2-home\") pod \"cd6d46ec-da0d-450a-b83e-d25f47f9add2\" (UID: \"cd6d46ec-da0d-450a-b83e-d25f47f9add2\") "
Jul 14 12:23:44.114774 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:44.114610    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd6d46ec-da0d-450a-b83e-d25f47f9add2-tmp-dir\") pod \"cd6d46ec-da0d-450a-b83e-d25f47f9add2\" (UID: \"cd6d46ec-da0d-450a-b83e-d25f47f9add2\") "
Jul 14 12:23:44.114774 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:44.114657    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd6d46ec-da0d-450a-b83e-d25f47f9add2-dshm\") pod \"cd6d46ec-da0d-450a-b83e-d25f47f9add2\" (UID: \"cd6d46ec-da0d-450a-b83e-d25f47f9add2\") "
Jul 14 12:23:44.114973 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:44.114820    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd6d46ec-da0d-450a-b83e-d25f47f9add2-home" (OuterVolumeSpecName: "home") pod "cd6d46ec-da0d-450a-b83e-d25f47f9add2" (UID: "cd6d46ec-da0d-450a-b83e-d25f47f9add2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 12:23:44.114973 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:44.114864    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd6d46ec-da0d-450a-b83e-d25f47f9add2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cd6d46ec-da0d-450a-b83e-d25f47f9add2" (UID: "cd6d46ec-da0d-450a-b83e-d25f47f9add2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 12:23:44.114973 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:44.114849    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd6d46ec-da0d-450a-b83e-d25f47f9add2-model-cache" (OuterVolumeSpecName: "model-cache") pod "cd6d46ec-da0d-450a-b83e-d25f47f9add2" (UID: "cd6d46ec-da0d-450a-b83e-d25f47f9add2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 12:23:44.116789 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:44.116761    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd6d46ec-da0d-450a-b83e-d25f47f9add2-dshm" (OuterVolumeSpecName: "dshm") pod "cd6d46ec-da0d-450a-b83e-d25f47f9add2" (UID: "cd6d46ec-da0d-450a-b83e-d25f47f9add2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 12:23:44.116893 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:44.116795    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd6d46ec-da0d-450a-b83e-d25f47f9add2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cd6d46ec-da0d-450a-b83e-d25f47f9add2" (UID: "cd6d46ec-da0d-450a-b83e-d25f47f9add2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 12:23:44.116893 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:44.116809    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd6d46ec-da0d-450a-b83e-d25f47f9add2-kube-api-access-8ldbh" (OuterVolumeSpecName: "kube-api-access-8ldbh") pod "cd6d46ec-da0d-450a-b83e-d25f47f9add2" (UID: "cd6d46ec-da0d-450a-b83e-d25f47f9add2"). InnerVolumeSpecName "kube-api-access-8ldbh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 12:23:44.215338 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:44.215233    2551 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd6d46ec-da0d-450a-b83e-d25f47f9add2-home\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 12:23:44.215338 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:44.215270    2551 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd6d46ec-da0d-450a-b83e-d25f47f9add2-tmp-dir\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 12:23:44.215338 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:44.215280    2551 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd6d46ec-da0d-450a-b83e-d25f47f9add2-dshm\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 12:23:44.215338 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:44.215304    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ldbh\" (UniqueName: \"kubernetes.io/projected/cd6d46ec-da0d-450a-b83e-d25f47f9add2-kube-api-access-8ldbh\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 12:23:44.215338 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:44.215315    2551 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd6d46ec-da0d-450a-b83e-d25f47f9add2-model-cache\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 12:23:44.215338 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:44.215323    2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd6d46ec-da0d-450a-b83e-d25f47f9add2-tls-certs\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 12:23:44.471348 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:44.471255    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7673-139638377596224"]
Jul 14 12:23:44.471636 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:44.471623    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="969427af-f358-4a9c-b2fb-dbdd7b80f7df" containerName="main"
Jul 14 12:23:44.471680 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:44.471638    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="969427af-f358-4a9c-b2fb-dbdd7b80f7df" containerName="main"
Jul 14 12:23:44.471680 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:44.471657    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd6d46ec-da0d-450a-b83e-d25f47f9add2" containerName="main"
Jul 14 12:23:44.471680 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:44.471663    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd6d46ec-da0d-450a-b83e-d25f47f9add2" containerName="main"
Jul 14 12:23:44.471798 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:44.471717    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd6d46ec-da0d-450a-b83e-d25f47f9add2" containerName="main"
Jul 14 12:23:44.471798 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:44.471728    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="969427af-f358-4a9c-b2fb-dbdd7b80f7df" containerName="main"
Jul 14 12:23:44.474867 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:44.474849    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7673-139638377596224"
Jul 14 12:23:44.478058 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:44.478025    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-dockercfg-gl5s8\""
Jul 14 12:23:44.484310 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:44.484282    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7673-139638377596224"]
Jul 14 12:23:44.619345 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:44.619301    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xl6f4\" (UniqueName: \"kubernetes.io/projected/182e9e03-e1ad-40ab-8d94-4bece74830d9-kube-api-access-xl6f4\") pod \"test-curl-7673-139638377596224\" (UID: \"182e9e03-e1ad-40ab-8d94-4bece74830d9\") " pod="openshift-ingress/test-curl-7673-139638377596224"
Jul 14 12:23:44.720885 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:44.720840    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xl6f4\" (UniqueName: \"kubernetes.io/projected/182e9e03-e1ad-40ab-8d94-4bece74830d9-kube-api-access-xl6f4\") pod \"test-curl-7673-139638377596224\" (UID: \"182e9e03-e1ad-40ab-8d94-4bece74830d9\") " pod="openshift-ingress/test-curl-7673-139638377596224"
Jul 14 12:23:44.732839 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:44.732762    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xl6f4\" (UniqueName: \"kubernetes.io/projected/182e9e03-e1ad-40ab-8d94-4bece74830d9-kube-api-access-xl6f4\") pod \"test-curl-7673-139638377596224\" (UID: \"182e9e03-e1ad-40ab-8d94-4bece74830d9\") " pod="openshift-ingress/test-curl-7673-139638377596224"
Jul 14 12:23:44.784935 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:44.784895    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7673-139638377596224"
Jul 14 12:23:44.909541 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:44.909489    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-480a7f/rate-test-model-480a7f-kserve-7cb6d59f47-76qqt"
Jul 14 12:23:44.910019 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:44.909487    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-480a7f/rate-test-model-480a7f-kserve-7cb6d59f47-76qqt" event={"ID":"cd6d46ec-da0d-450a-b83e-d25f47f9add2","Type":"ContainerDied","Data":"b87486d5d395ef1e0f72422f02a8b7529085b0169a0d0dbc93aa588788a6b399"}
Jul 14 12:23:44.910019 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:44.909655    2551 scope.go:117] "RemoveContainer" containerID="deb1ab944e35dd37c516efe8bc8e9c115c2a7493803c3a009884ace77abb1c56"
Jul 14 12:23:44.929218 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:44.929191    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7673-139638377596224"]
Jul 14 12:23:44.932134 ip-10-0-136-173 kubenswrapper[2551]: W0714 12:23:44.932105    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod182e9e03_e1ad_40ab_8d94_4bece74830d9.slice/crio-9fec3def289a28082c717dfaabfb73f42a8c71d062ef946a1ef1fe2a47ff4433 WatchSource:0}: Error finding container 9fec3def289a28082c717dfaabfb73f42a8c71d062ef946a1ef1fe2a47ff4433: Status 404 returned error can't find the container with id 9fec3def289a28082c717dfaabfb73f42a8c71d062ef946a1ef1fe2a47ff4433
Jul 14 12:23:44.933521 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:44.933500    2551 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 14 12:23:44.936604 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:44.936566    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-480a7f/rate-test-model-480a7f-kserve-7cb6d59f47-76qqt"]
Jul 14 12:23:44.941746 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:44.941723    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-480a7f/rate-test-model-480a7f-kserve-7cb6d59f47-76qqt"]
Jul 14 12:23:45.191762 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:45.191725    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="969427af-f358-4a9c-b2fb-dbdd7b80f7df" path="/var/lib/kubelet/pods/969427af-f358-4a9c-b2fb-dbdd7b80f7df/volumes"
Jul 14 12:23:45.192133 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:45.192120    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd6d46ec-da0d-450a-b83e-d25f47f9add2" path="/var/lib/kubelet/pods/cd6d46ec-da0d-450a-b83e-d25f47f9add2/volumes"
Jul 14 12:23:45.917321 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:45.917222    2551 generic.go:356] "Generic (PLEG): container finished" podID="182e9e03-e1ad-40ab-8d94-4bece74830d9" containerID="b7fdd2cbf9c5ffd2a3bb5d39901efbf6db79b644cf49c07e4ef05cc720a60f7c" exitCode=0
Jul 14 12:23:45.917757 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:45.917308    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7673-139638377596224" event={"ID":"182e9e03-e1ad-40ab-8d94-4bece74830d9","Type":"ContainerDied","Data":"b7fdd2cbf9c5ffd2a3bb5d39901efbf6db79b644cf49c07e4ef05cc720a60f7c"}
Jul 14 12:23:45.917757 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:45.917356    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7673-139638377596224" event={"ID":"182e9e03-e1ad-40ab-8d94-4bece74830d9","Type":"ContainerStarted","Data":"9fec3def289a28082c717dfaabfb73f42a8c71d062ef946a1ef1fe2a47ff4433"}
Jul 14 12:23:47.071595 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:47.071551    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7673-139638377596224"
Jul 14 12:23:47.100109 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:47.100070    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7673-139638377596224_182e9e03-e1ad-40ab-8d94-4bece74830d9/test-curl-7673-139638377596224/0.log"
Jul 14 12:23:47.130081 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:47.130044    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7673-139638377596224"]
Jul 14 12:23:47.137990 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:47.137958    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7673-139638377596224"]
Jul 14 12:23:47.140352 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:47.140332    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xl6f4\" (UniqueName: \"kubernetes.io/projected/182e9e03-e1ad-40ab-8d94-4bece74830d9-kube-api-access-xl6f4\") pod \"182e9e03-e1ad-40ab-8d94-4bece74830d9\" (UID: \"182e9e03-e1ad-40ab-8d94-4bece74830d9\") "
Jul 14 12:23:47.142383 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:47.142352    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/182e9e03-e1ad-40ab-8d94-4bece74830d9-kube-api-access-xl6f4" (OuterVolumeSpecName: "kube-api-access-xl6f4") pod "182e9e03-e1ad-40ab-8d94-4bece74830d9" (UID: "182e9e03-e1ad-40ab-8d94-4bece74830d9"). InnerVolumeSpecName "kube-api-access-xl6f4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 12:23:47.191572 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:47.191536    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="182e9e03-e1ad-40ab-8d94-4bece74830d9" path="/var/lib/kubelet/pods/182e9e03-e1ad-40ab-8d94-4bece74830d9/volumes"
Jul 14 12:23:47.241790 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:47.241749    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xl6f4\" (UniqueName: \"kubernetes.io/projected/182e9e03-e1ad-40ab-8d94-4bece74830d9-kube-api-access-xl6f4\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 12:23:47.930119 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:47.930089    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7673-139638377596224"
Jul 14 12:23:47.930309 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:47.930142    2551 scope.go:117] "RemoveContainer" containerID="b7fdd2cbf9c5ffd2a3bb5d39901efbf6db79b644cf49c07e4ef05cc720a60f7c"
Jul 14 12:23:47.976745 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:47.976710    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:23:48.075427 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:48.075388    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:23:57.883879 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:57.883786    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-cac82f-7b45ddd4fc-85j8h"]
Jul 14 12:23:57.884698 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:57.884639    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-cac82f-7b45ddd4fc-85j8h" podUID="8efbff05-6d0b-4f2a-be7b-84126004eee8" containerName="maas-api" containerID="cri-o://41cf58bd3a7c1cef31206a755bbd7d79647b0c526c60869b36a533ffa89494b4" gracePeriod=30
Jul 14 12:23:57.908701 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:57.908662    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:23:58.154941 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:58.154915    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-cac82f-7b45ddd4fc-85j8h"
Jul 14 12:23:58.238286 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:58.238251    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8efbff05-6d0b-4f2a-be7b-84126004eee8-maas-api-tls\") pod \"8efbff05-6d0b-4f2a-be7b-84126004eee8\" (UID: \"8efbff05-6d0b-4f2a-be7b-84126004eee8\") "
Jul 14 12:23:58.238489 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:58.238314    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8swsv\" (UniqueName: \"kubernetes.io/projected/8efbff05-6d0b-4f2a-be7b-84126004eee8-kube-api-access-8swsv\") pod \"8efbff05-6d0b-4f2a-be7b-84126004eee8\" (UID: \"8efbff05-6d0b-4f2a-be7b-84126004eee8\") "
Jul 14 12:23:58.238489 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:58.238422    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8efbff05-6d0b-4f2a-be7b-84126004eee8-openshift-service-ca\") pod \"8efbff05-6d0b-4f2a-be7b-84126004eee8\" (UID: \"8efbff05-6d0b-4f2a-be7b-84126004eee8\") "
Jul 14 12:23:58.238805 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:58.238775    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8efbff05-6d0b-4f2a-be7b-84126004eee8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8efbff05-6d0b-4f2a-be7b-84126004eee8" (UID: "8efbff05-6d0b-4f2a-be7b-84126004eee8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 12:23:58.240498 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:58.240470    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8efbff05-6d0b-4f2a-be7b-84126004eee8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8efbff05-6d0b-4f2a-be7b-84126004eee8" (UID: "8efbff05-6d0b-4f2a-be7b-84126004eee8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 12:23:58.240498 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:58.240483    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8efbff05-6d0b-4f2a-be7b-84126004eee8-kube-api-access-8swsv" (OuterVolumeSpecName: "kube-api-access-8swsv") pod "8efbff05-6d0b-4f2a-be7b-84126004eee8" (UID: "8efbff05-6d0b-4f2a-be7b-84126004eee8"). InnerVolumeSpecName "kube-api-access-8swsv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 12:23:58.339791 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:58.339751    2551 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8efbff05-6d0b-4f2a-be7b-84126004eee8-maas-api-tls\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 12:23:58.339791 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:58.339787    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8swsv\" (UniqueName: \"kubernetes.io/projected/8efbff05-6d0b-4f2a-be7b-84126004eee8-kube-api-access-8swsv\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 12:23:58.339791 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:58.339797    2551 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8efbff05-6d0b-4f2a-be7b-84126004eee8-openshift-service-ca\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 12:23:58.985327 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:58.985284    2551 generic.go:356] "Generic (PLEG): container finished" podID="8efbff05-6d0b-4f2a-be7b-84126004eee8" containerID="41cf58bd3a7c1cef31206a755bbd7d79647b0c526c60869b36a533ffa89494b4" exitCode=0
Jul 14 12:23:58.985765 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:58.985348    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-cac82f-7b45ddd4fc-85j8h"
Jul 14 12:23:58.985765 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:58.985364    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-cac82f-7b45ddd4fc-85j8h" event={"ID":"8efbff05-6d0b-4f2a-be7b-84126004eee8","Type":"ContainerDied","Data":"41cf58bd3a7c1cef31206a755bbd7d79647b0c526c60869b36a533ffa89494b4"}
Jul 14 12:23:58.985765 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:58.985466    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-cac82f-7b45ddd4fc-85j8h" event={"ID":"8efbff05-6d0b-4f2a-be7b-84126004eee8","Type":"ContainerDied","Data":"c78a905f505932cddf3a22a6a36d909e569e68a9378f377fa2d356dbe7a7eb32"}
Jul 14 12:23:58.985765 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:58.985497    2551 scope.go:117] "RemoveContainer" containerID="41cf58bd3a7c1cef31206a755bbd7d79647b0c526c60869b36a533ffa89494b4"
Jul 14 12:23:58.994537 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:58.994515    2551 scope.go:117] "RemoveContainer" containerID="41cf58bd3a7c1cef31206a755bbd7d79647b0c526c60869b36a533ffa89494b4"
Jul 14 12:23:58.994854 ip-10-0-136-173 kubenswrapper[2551]: E0714 12:23:58.994830    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41cf58bd3a7c1cef31206a755bbd7d79647b0c526c60869b36a533ffa89494b4\": container with ID starting with 41cf58bd3a7c1cef31206a755bbd7d79647b0c526c60869b36a533ffa89494b4 not found: ID does not exist" containerID="41cf58bd3a7c1cef31206a755bbd7d79647b0c526c60869b36a533ffa89494b4"
Jul 14 12:23:58.994901 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:58.994865    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41cf58bd3a7c1cef31206a755bbd7d79647b0c526c60869b36a533ffa89494b4"} err="failed to get container status \"41cf58bd3a7c1cef31206a755bbd7d79647b0c526c60869b36a533ffa89494b4\": rpc error: code = NotFound desc = could not find container \"41cf58bd3a7c1cef31206a755bbd7d79647b0c526c60869b36a533ffa89494b4\": container with ID starting with 41cf58bd3a7c1cef31206a755bbd7d79647b0c526c60869b36a533ffa89494b4 not found: ID does not exist"
Jul 14 12:23:59.010376 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:59.010343    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-cac82f-7b45ddd4fc-85j8h"]
Jul 14 12:23:59.016447 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:59.016423    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-cac82f-7b45ddd4fc-85j8h"]
Jul 14 12:23:59.192698 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:23:59.192655    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8efbff05-6d0b-4f2a-be7b-84126004eee8" path="/var/lib/kubelet/pods/8efbff05-6d0b-4f2a-be7b-84126004eee8/volumes"
Jul 14 12:24:01.595061 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:24:01.595024    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 12:24:02.456928 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:24:02.456898    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:24:09.710213 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:24:09.710175    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:24:09.810445 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:24:09.810408    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:24:13.316197 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:24:13.316154    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:24:13.516234 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:24:13.516194    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:24:14.260748 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:24:14.260712    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:24:14.365151 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:24:14.365112    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:24:24.183370 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:24:24.183332    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-480a7f-5f4888d679-8vzkj"]
Jul 14 12:24:24.183852 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:24:24.183563    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-480a7f-5f4888d679-8vzkj" podUID="91892839-6603-4e49-a3b3-6ab94ca8ae17" containerName="maas-api" containerID="cri-o://6cac5c700384e34064250de15ba580956acfb3a27b1435d97f363072798dbe1b" gracePeriod=30
Jul 14 12:24:24.222887 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:24:24.222856    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:24:24.440783 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:24:24.440754    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-480a7f-5f4888d679-8vzkj"
Jul 14 12:24:24.570374 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:24:24.570328    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91892839-6603-4e49-a3b3-6ab94ca8ae17-openshift-service-ca\") pod \"91892839-6603-4e49-a3b3-6ab94ca8ae17\" (UID: \"91892839-6603-4e49-a3b3-6ab94ca8ae17\") "
Jul 14 12:24:24.570602 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:24:24.570416    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k5lc2\" (UniqueName: \"kubernetes.io/projected/91892839-6603-4e49-a3b3-6ab94ca8ae17-kube-api-access-k5lc2\") pod \"91892839-6603-4e49-a3b3-6ab94ca8ae17\" (UID: \"91892839-6603-4e49-a3b3-6ab94ca8ae17\") "
Jul 14 12:24:24.570602 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:24:24.570506    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91892839-6603-4e49-a3b3-6ab94ca8ae17-maas-api-tls\") pod \"91892839-6603-4e49-a3b3-6ab94ca8ae17\" (UID: \"91892839-6603-4e49-a3b3-6ab94ca8ae17\") "
Jul 14 12:24:24.570853 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:24:24.570822    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91892839-6603-4e49-a3b3-6ab94ca8ae17-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "91892839-6603-4e49-a3b3-6ab94ca8ae17" (UID: "91892839-6603-4e49-a3b3-6ab94ca8ae17"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 12:24:24.572632 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:24:24.572601    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91892839-6603-4e49-a3b3-6ab94ca8ae17-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "91892839-6603-4e49-a3b3-6ab94ca8ae17" (UID: "91892839-6603-4e49-a3b3-6ab94ca8ae17"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 12:24:24.572752 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:24:24.572679    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91892839-6603-4e49-a3b3-6ab94ca8ae17-kube-api-access-k5lc2" (OuterVolumeSpecName: "kube-api-access-k5lc2") pod "91892839-6603-4e49-a3b3-6ab94ca8ae17" (UID: "91892839-6603-4e49-a3b3-6ab94ca8ae17"). InnerVolumeSpecName "kube-api-access-k5lc2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 12:24:24.671356 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:24:24.671316    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5lc2\" (UniqueName: \"kubernetes.io/projected/91892839-6603-4e49-a3b3-6ab94ca8ae17-kube-api-access-k5lc2\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 12:24:24.671356 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:24:24.671352    2551 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91892839-6603-4e49-a3b3-6ab94ca8ae17-maas-api-tls\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 12:24:24.671356 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:24:24.671365    2551 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91892839-6603-4e49-a3b3-6ab94ca8ae17-openshift-service-ca\") on node \"ip-10-0-136-173.ec2.internal\" DevicePath \"\""
Jul 14 12:24:25.111423 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:24:25.111386    2551 generic.go:356] "Generic (PLEG): container finished" podID="91892839-6603-4e49-a3b3-6ab94ca8ae17" containerID="6cac5c700384e34064250de15ba580956acfb3a27b1435d97f363072798dbe1b" exitCode=0
Jul 14 12:24:25.111640 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:24:25.111438    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-480a7f-5f4888d679-8vzkj"
Jul 14 12:24:25.111640 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:24:25.111471    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-480a7f-5f4888d679-8vzkj" event={"ID":"91892839-6603-4e49-a3b3-6ab94ca8ae17","Type":"ContainerDied","Data":"6cac5c700384e34064250de15ba580956acfb3a27b1435d97f363072798dbe1b"}
Jul 14 12:24:25.111640 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:24:25.111528    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-480a7f-5f4888d679-8vzkj" event={"ID":"91892839-6603-4e49-a3b3-6ab94ca8ae17","Type":"ContainerDied","Data":"afc17bb6656956d71110449e07d5512e4f962df42f2d96a8eb1ee937d0e0257c"}
Jul 14 12:24:25.111640 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:24:25.111551    2551 scope.go:117] "RemoveContainer" containerID="6cac5c700384e34064250de15ba580956acfb3a27b1435d97f363072798dbe1b"
Jul 14 12:24:25.121474 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:24:25.121447    2551 scope.go:117] "RemoveContainer" containerID="6cac5c700384e34064250de15ba580956acfb3a27b1435d97f363072798dbe1b"
Jul 14 12:24:25.121980 ip-10-0-136-173 kubenswrapper[2551]: E0714 12:24:25.121914    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cac5c700384e34064250de15ba580956acfb3a27b1435d97f363072798dbe1b\": container with ID starting with 6cac5c700384e34064250de15ba580956acfb3a27b1435d97f363072798dbe1b not found: ID does not exist" containerID="6cac5c700384e34064250de15ba580956acfb3a27b1435d97f363072798dbe1b"
Jul 14 12:24:25.121980 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:24:25.121954    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cac5c700384e34064250de15ba580956acfb3a27b1435d97f363072798dbe1b"} err="failed to get container status \"6cac5c700384e34064250de15ba580956acfb3a27b1435d97f363072798dbe1b\": rpc error: code = NotFound desc = could not find container \"6cac5c700384e34064250de15ba580956acfb3a27b1435d97f363072798dbe1b\": container with ID starting with 6cac5c700384e34064250de15ba580956acfb3a27b1435d97f363072798dbe1b not found: ID does not exist"
Jul 14 12:24:25.141303 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:24:25.139689    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-480a7f-5f4888d679-8vzkj"]
Jul 14 12:24:25.146836 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:24:25.146809    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-480a7f-5f4888d679-8vzkj"]
Jul 14 12:24:25.190884 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:24:25.190851    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91892839-6603-4e49-a3b3-6ab94ca8ae17" path="/var/lib/kubelet/pods/91892839-6603-4e49-a3b3-6ab94ca8ae17/volumes"
Jul 14 12:24:27.301145 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:24:27.301110    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7pqvf"]
Jul 14 12:24:28.748134 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:24:28.748098    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:24:36.413215 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:24:36.413174    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:24:36.510683 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:24:36.510651    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:24:39.559909 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:24:39.559869    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:24:39.786717 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:24:39.786687    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:24:45.299297 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:24:45.299263    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-gl6pr_1540b9c9-a79b-47b4-8183-ea78634934db/manager/0.log"
Jul 14 12:24:45.422313 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:24:45.422283    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-zqj74_ea5ed537-f005-4a9d-8bd2-34dfe38244b1/manager/0.log"
Jul 14 12:24:45.664075 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:24:45.663989    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-ggvrl_dc2f63ce-780c-454a-a5a1-4827c3f08fd8/server/0.log"
Jul 14 12:24:45.794343 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:24:45.794312    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-b9lvb_702307ef-b102-4d06-8246-c81e31a73276/manager/0.log"
Jul 14 12:24:46.179786 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:24:46.179759    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6c944d8cbf-xl5vp_13561ca4-eb2a-4cc6-9abd-3f8ce624cbc9/copy-manifests/0.log"
Jul 14 12:24:46.195041 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:24:46.195003    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6c944d8cbf-xl5vp_13561ca4-eb2a-4cc6-9abd-3f8ce624cbc9/manager/0.log"
Jul 14 12:24:49.869179 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:24:49.869134    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-m7jvb_b0531370-9f45-44e1-8e67-f32852cd15a9/manager/0.log"
Jul 14 12:24:49.981501 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:24:49.981467    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-9cpbg_2ead8ac4-7d3e-4cff-8264-7e3229692eac/kuadrant-console-plugin/0.log"
Jul 14 12:24:50.114244 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:24:50.114211    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-99d4p_340cc9d9-9afb-4f6e-b35d-3c4ceea20c83/registry-server/0.log"
Jul 14 12:24:50.245238 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:24:50.245193    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-fnhg4_85a8f743-50a7-4e22-8fd5-585aa4c9d2d9/manager/0.log"
Jul 14 12:24:50.361449 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:24:50.361421    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-7pqvf_92a411dc-0055-4a4c-8302-5ffc7ceff6a0/limitador/0.log"
Jul 14 12:24:50.485515 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:24:50.485483    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-cqrmp_3f3d650b-412d-4402-853e-e3b9b752a0ac/manager/0.log"
Jul 14 12:24:51.081019 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:24:51.080985    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7d55c74c4b-mhfwz_341d716b-2dda-4d65-83d1-a2b4e64ae139/kube-auth-proxy/0.log"
Jul 14 12:24:51.324901 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:24:51.324862    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-r7zl5_2b49d636-77b9-487e-aa76-05a41fe47121/istio-proxy/1.log"
Jul 14 12:24:52.140962 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:24:52.140933    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-9nnlc_74e33bbf-9897-4a16-a7e5-5a2bfcc7518a/main/0.log"
Jul 14 12:24:52.393120 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:24:52.393016    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-tz94n_3d190ee5-ef81-4dad-9661-80b09580d03d/main/0.log"
Jul 14 12:24:56.844370 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:24:56.844334    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-t6fcf/must-gather-4pj6p"]
Jul 14 12:24:56.844770 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:24:56.844726    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91892839-6603-4e49-a3b3-6ab94ca8ae17" containerName="maas-api"
Jul 14 12:24:56.844770 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:24:56.844740    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="91892839-6603-4e49-a3b3-6ab94ca8ae17" containerName="maas-api"
Jul 14 12:24:56.844770 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:24:56.844748    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8efbff05-6d0b-4f2a-be7b-84126004eee8" containerName="maas-api"
Jul 14 12:24:56.844770 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:24:56.844754    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="8efbff05-6d0b-4f2a-be7b-84126004eee8" containerName="maas-api"
Jul 14 12:24:56.844770 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:24:56.844762    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="182e9e03-e1ad-40ab-8d94-4bece74830d9" containerName="test-curl-7673-139638377596224"
Jul 14 12:24:56.844770 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:24:56.844769    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="182e9e03-e1ad-40ab-8d94-4bece74830d9" containerName="test-curl-7673-139638377596224"
Jul 14 12:24:56.844960 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:24:56.844839    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="91892839-6603-4e49-a3b3-6ab94ca8ae17" containerName="maas-api"
Jul 14 12:24:56.844960 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:24:56.844848    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="182e9e03-e1ad-40ab-8d94-4bece74830d9" containerName="test-curl-7673-139638377596224"
Jul 14 12:24:56.844960 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:24:56.844855    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="8efbff05-6d0b-4f2a-be7b-84126004eee8" containerName="maas-api"
Jul 14 12:24:56.850664 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:24:56.848999    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-t6fcf/must-gather-4pj6p"
Jul 14 12:24:56.852393 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:24:56.852366    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-t6fcf\"/\"kube-root-ca.crt\""
Jul 14 12:24:56.852526 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:24:56.852400    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-t6fcf\"/\"default-dockercfg-tcbhs\""
Jul 14 12:24:56.852526 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:24:56.852377    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-t6fcf\"/\"openshift-service-ca.crt\""
Jul 14 12:24:56.859936 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:24:56.859911    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-t6fcf/must-gather-4pj6p"]
Jul 14 12:24:56.875209 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:24:56.875176    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:24:56.951681 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:24:56.951635    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f0ef184d-b022-41e4-804c-c8dddc7441e0-must-gather-output\") pod \"must-gather-4pj6p\" (UID: \"f0ef184d-b022-41e4-804c-c8dddc7441e0\") " pod="openshift-must-gather-t6fcf/must-gather-4pj6p"
Jul 14 12:24:56.951681 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:24:56.951680    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99949\" (UniqueName: \"kubernetes.io/projected/f0ef184d-b022-41e4-804c-c8dddc7441e0-kube-api-access-99949\") pod \"must-gather-4pj6p\" (UID: \"f0ef184d-b022-41e4-804c-c8dddc7441e0\") " pod="openshift-must-gather-t6fcf/must-gather-4pj6p"
Jul 14 12:24:56.962294 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:24:56.962267    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:24:57.052533 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:24:57.052477    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f0ef184d-b022-41e4-804c-c8dddc7441e0-must-gather-output\") pod \"must-gather-4pj6p\" (UID: \"f0ef184d-b022-41e4-804c-c8dddc7441e0\") " pod="openshift-must-gather-t6fcf/must-gather-4pj6p"
Jul 14 12:24:57.052533 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:24:57.052536    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99949\" (UniqueName: \"kubernetes.io/projected/f0ef184d-b022-41e4-804c-c8dddc7441e0-kube-api-access-99949\") pod \"must-gather-4pj6p\" (UID: \"f0ef184d-b022-41e4-804c-c8dddc7441e0\") " pod="openshift-must-gather-t6fcf/must-gather-4pj6p"
Jul 14 12:24:57.052883 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:24:57.052862    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f0ef184d-b022-41e4-804c-c8dddc7441e0-must-gather-output\") pod \"must-gather-4pj6p\" (UID: \"f0ef184d-b022-41e4-804c-c8dddc7441e0\") " pod="openshift-must-gather-t6fcf/must-gather-4pj6p"
Jul 14 12:24:57.062855 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:24:57.062820    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-99949\" (UniqueName: \"kubernetes.io/projected/f0ef184d-b022-41e4-804c-c8dddc7441e0-kube-api-access-99949\") pod \"must-gather-4pj6p\" (UID: \"f0ef184d-b022-41e4-804c-c8dddc7441e0\") " pod="openshift-must-gather-t6fcf/must-gather-4pj6p"
Jul 14 12:24:57.161974 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:24:57.161882    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-t6fcf/must-gather-4pj6p"
Jul 14 12:24:57.316618 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:24:57.316591    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-t6fcf/must-gather-4pj6p"]
Jul 14 12:24:57.318411 ip-10-0-136-173 kubenswrapper[2551]: W0714 12:24:57.318375    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf0ef184d_b022_41e4_804c_c8dddc7441e0.slice/crio-5e35f423ede1369d3fa3cfcf5816fb99e958d68ab769192ddd7386c154679da6 WatchSource:0}: Error finding container 5e35f423ede1369d3fa3cfcf5816fb99e958d68ab769192ddd7386c154679da6: Status 404 returned error can't find the container with id 5e35f423ede1369d3fa3cfcf5816fb99e958d68ab769192ddd7386c154679da6
Jul 14 12:24:58.274725 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:24:58.274683    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-t6fcf/must-gather-4pj6p" event={"ID":"f0ef184d-b022-41e4-804c-c8dddc7441e0","Type":"ContainerStarted","Data":"5e35f423ede1369d3fa3cfcf5816fb99e958d68ab769192ddd7386c154679da6"}
Jul 14 12:24:58.351798 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:24:58.351764    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log" limit=1024
Jul 14 12:24:59.281623 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:24:59.281568    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-t6fcf/must-gather-4pj6p" event={"ID":"f0ef184d-b022-41e4-804c-c8dddc7441e0","Type":"ContainerStarted","Data":"c3875c64aba52ea0a204312b84a9d651b1761af9854ae80763578697fa61633c"}
Jul 14 12:24:59.282107 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:24:59.281634    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-t6fcf/must-gather-4pj6p" event={"ID":"f0ef184d-b022-41e4-804c-c8dddc7441e0","Type":"ContainerStarted","Data":"53d37863fb6292827fae032f909c1740dbcd8ccc7557c04bcb59fd139a322dad"}
Jul 14 12:24:59.314753 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:24:59.314680    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-t6fcf/must-gather-4pj6p" podStartSLOduration=2.344440009 podStartE2EDuration="3.314654129s" totalImagesPullingTime="970.21412ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 12:24:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 12:24:59.312822264 +0000 UTC m=+2850.544631559" watchObservedRunningTime="2026-07-14 12:24:59.314654129 +0000 UTC m=+2850.546463426"
Jul 14 12:25:00.118794 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:25:00.118718    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-75pwf_3360b04a-23e1-49f9-84e4-11186ccde5e7/global-pull-secret-syncer/0.log"
Jul 14 12:25:00.384551 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:25:00.384454    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-642k8_8a3aafb3-0f38-444f-98c4-dbf37fa6effe/konnectivity-agent/0.log"
Jul 14 12:25:00.510483 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:25:00.510451    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-173.ec2.internal_a552e96d3a56282f33a24bc1bc8a195b/haproxy/0.log"
Jul 14 12:25:05.863364 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:25:05.863301    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-m7jvb_b0531370-9f45-44e1-8e67-f32852cd15a9/manager/0.log"
Jul 14 12:25:05.904001 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:25:05.903781    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-9cpbg_2ead8ac4-7d3e-4cff-8264-7e3229692eac/kuadrant-console-plugin/0.log"
Jul 14 12:25:05.972503 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:25:05.972342    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-99d4p_340cc9d9-9afb-4f6e-b35d-3c4ceea20c83/registry-server/0.log"
Jul 14 12:25:06.155121 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:25:06.154996    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-fnhg4_85a8f743-50a7-4e22-8fd5-585aa4c9d2d9/manager/0.log"
Jul 14 12:25:06.180392 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:25:06.180272    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-7pqvf_92a411dc-0055-4a4c-8302-5ffc7ceff6a0/limitador/0.log"
Jul 14 12:25:06.271787 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:25:06.271735    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-cqrmp_3f3d650b-412d-4402-853e-e3b9b752a0ac/manager/0.log"
Jul 14 12:25:08.674979 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:25:08.674949    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lht2c_8aa81994-2723-49fc-af2f-a832f71b0973/node-exporter/0.log"
Jul 14 12:25:08.704112 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:25:08.704079    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lht2c_8aa81994-2723-49fc-af2f-a832f71b0973/kube-rbac-proxy/0.log"
Jul 14 12:25:08.744227 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:25:08.744199    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lht2c_8aa81994-2723-49fc-af2f-a832f71b0973/init-textfile/0.log"
Jul 14 12:25:10.223078 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:25:10.223020    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-t6fcf/perf-node-gather-daemonset-djh9z"]
Jul 14 12:25:10.231238 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:25:10.231202    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-t6fcf/perf-node-gather-daemonset-djh9z"
Jul 14 12:25:10.241029 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:25:10.240993    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-t6fcf/perf-node-gather-daemonset-djh9z"]
Jul 14 12:25:10.404425 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:25:10.404352    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e27437d5-d5c3-4129-a175-72a013da03c5-proc\") pod \"perf-node-gather-daemonset-djh9z\" (UID: \"e27437d5-d5c3-4129-a175-72a013da03c5\") " pod="openshift-must-gather-t6fcf/perf-node-gather-daemonset-djh9z"
Jul 14 12:25:10.404737 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:25:10.404715    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e27437d5-d5c3-4129-a175-72a013da03c5-lib-modules\") pod \"perf-node-gather-daemonset-djh9z\" (UID: \"e27437d5-d5c3-4129-a175-72a013da03c5\") " pod="openshift-must-gather-t6fcf/perf-node-gather-daemonset-djh9z"
Jul 14 12:25:10.404889 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:25:10.404876    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e27437d5-d5c3-4129-a175-72a013da03c5-podres\") pod \"perf-node-gather-daemonset-djh9z\" (UID: \"e27437d5-d5c3-4129-a175-72a013da03c5\") " pod="openshift-must-gather-t6fcf/perf-node-gather-daemonset-djh9z"
Jul 14 12:25:10.405024 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:25:10.405011    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tj6vl\" (UniqueName: \"kubernetes.io/projected/e27437d5-d5c3-4129-a175-72a013da03c5-kube-api-access-tj6vl\") pod \"perf-node-gather-daemonset-djh9z\" (UID: \"e27437d5-d5c3-4129-a175-72a013da03c5\") " pod="openshift-must-gather-t6fcf/perf-node-gather-daemonset-djh9z"
Jul 14 12:25:10.405182 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:25:10.405170    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e27437d5-d5c3-4129-a175-72a013da03c5-sys\") pod \"perf-node-gather-daemonset-djh9z\" (UID: \"e27437d5-d5c3-4129-a175-72a013da03c5\") " pod="openshift-must-gather-t6fcf/perf-node-gather-daemonset-djh9z"
Jul 14 12:25:10.505835 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:25:10.505733    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e27437d5-d5c3-4129-a175-72a013da03c5-sys\") pod \"perf-node-gather-daemonset-djh9z\" (UID: \"e27437d5-d5c3-4129-a175-72a013da03c5\") " pod="openshift-must-gather-t6fcf/perf-node-gather-daemonset-djh9z"
Jul 14 12:25:10.505835 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:25:10.505806    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e27437d5-d5c3-4129-a175-72a013da03c5-proc\") pod \"perf-node-gather-daemonset-djh9z\" (UID: \"e27437d5-d5c3-4129-a175-72a013da03c5\") " pod="openshift-must-gather-t6fcf/perf-node-gather-daemonset-djh9z"
Jul 14 12:25:10.505835 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:25:10.505828    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e27437d5-d5c3-4129-a175-72a013da03c5-lib-modules\") pod \"perf-node-gather-daemonset-djh9z\" (UID: \"e27437d5-d5c3-4129-a175-72a013da03c5\") " pod="openshift-must-gather-t6fcf/perf-node-gather-daemonset-djh9z"
Jul 14 12:25:10.506094 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:25:10.505845    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e27437d5-d5c3-4129-a175-72a013da03c5-podres\") pod \"perf-node-gather-daemonset-djh9z\" (UID: \"e27437d5-d5c3-4129-a175-72a013da03c5\") " pod="openshift-must-gather-t6fcf/perf-node-gather-daemonset-djh9z"
Jul 14 12:25:10.506094 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:25:10.505869    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tj6vl\" (UniqueName: \"kubernetes.io/projected/e27437d5-d5c3-4129-a175-72a013da03c5-kube-api-access-tj6vl\") pod \"perf-node-gather-daemonset-djh9z\" (UID: \"e27437d5-d5c3-4129-a175-72a013da03c5\") " pod="openshift-must-gather-t6fcf/perf-node-gather-daemonset-djh9z"
Jul 14 12:25:10.506094 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:25:10.505903    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e27437d5-d5c3-4129-a175-72a013da03c5-proc\") pod \"perf-node-gather-daemonset-djh9z\" (UID: \"e27437d5-d5c3-4129-a175-72a013da03c5\") " pod="openshift-must-gather-t6fcf/perf-node-gather-daemonset-djh9z"
Jul 14 12:25:10.506094 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:25:10.505985    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e27437d5-d5c3-4129-a175-72a013da03c5-sys\") pod \"perf-node-gather-daemonset-djh9z\" (UID: \"e27437d5-d5c3-4129-a175-72a013da03c5\") " pod="openshift-must-gather-t6fcf/perf-node-gather-daemonset-djh9z"
Jul 14 12:25:10.506094 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:25:10.506029    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e27437d5-d5c3-4129-a175-72a013da03c5-lib-modules\") pod \"perf-node-gather-daemonset-djh9z\" (UID: \"e27437d5-d5c3-4129-a175-72a013da03c5\") " pod="openshift-must-gather-t6fcf/perf-node-gather-daemonset-djh9z"
Jul 14 12:25:10.506304 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:25:10.506157    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e27437d5-d5c3-4129-a175-72a013da03c5-podres\") pod \"perf-node-gather-daemonset-djh9z\" (UID: \"e27437d5-d5c3-4129-a175-72a013da03c5\") " pod="openshift-must-gather-t6fcf/perf-node-gather-daemonset-djh9z"
Jul 14 12:25:10.516646 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:25:10.516616    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tj6vl\" (UniqueName: \"kubernetes.io/projected/e27437d5-d5c3-4129-a175-72a013da03c5-kube-api-access-tj6vl\") pod \"perf-node-gather-daemonset-djh9z\" (UID: \"e27437d5-d5c3-4129-a175-72a013da03c5\") " pod="openshift-must-gather-t6fcf/perf-node-gather-daemonset-djh9z"
Jul 14 12:25:10.549661 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:25:10.549614    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-t6fcf/perf-node-gather-daemonset-djh9z"
Jul 14 12:25:10.739231 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:25:10.738829    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-t6fcf/perf-node-gather-daemonset-djh9z"]
Jul 14 12:25:11.392817 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:25:11.392781    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-t6fcf/perf-node-gather-daemonset-djh9z" event={"ID":"e27437d5-d5c3-4129-a175-72a013da03c5","Type":"ContainerStarted","Data":"71cf968f42df0e5b6f579ea7513f5c7198d212c2aa0744e78622c2d24207665b"}
Jul 14 12:25:11.393343 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:25:11.392835    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-t6fcf/perf-node-gather-daemonset-djh9z" event={"ID":"e27437d5-d5c3-4129-a175-72a013da03c5","Type":"ContainerStarted","Data":"a7d724861ca7c1eab005597679f61699619da0d0f9a24fd97c4bb3998fc6973c"}
Jul 14 12:25:11.393343 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:25:11.392876    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-t6fcf/perf-node-gather-daemonset-djh9z"
Jul 14 12:25:11.432058 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:25:11.431998    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-t6fcf/perf-node-gather-daemonset-djh9z" podStartSLOduration=1.431982335 podStartE2EDuration="1.431982335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 12:25:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 12:25:11.431471441 +0000 UTC m=+2862.663280741" watchObservedRunningTime="2026-07-14 12:25:11.431982335 +0000 UTC m=+2862.663791630"
Jul 14 12:25:13.777237 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:25:13.777210    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5qmr8_4f703685-4285-43da-a074-362a513a32b1/dns/0.log"
Jul 14 12:25:13.808515 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:25:13.808483    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5qmr8_4f703685-4285-43da-a074-362a513a32b1/kube-rbac-proxy/0.log"
Jul 14 12:25:13.929963 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:25:13.929939    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mldq8_e356b532-03eb-4760-93eb-a731dbc0f142/dns-node-resolver/0.log"
Jul 14 12:25:14.584793 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:25:14.584767    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-27fdx_8590baa3-db08-49ef-8d26-3f551d6954e4/node-ca/0.log"
Jul 14 12:25:15.847485 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:25:15.847448    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7d55c74c4b-mhfwz_341d716b-2dda-4d65-83d1-a2b4e64ae139/kube-auth-proxy/0.log"
Jul 14 12:25:15.910750 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:25:15.910714    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-r7zl5_2b49d636-77b9-487e-aa76-05a41fe47121/istio-proxy/0.log"
Jul 14 12:25:15.958524 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:25:15.958477    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-r7zl5_2b49d636-77b9-487e-aa76-05a41fe47121/istio-proxy/1.log"
Jul 14 12:25:16.852596 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:25:16.852558    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-x8b5w_2585982a-d4d0-4d80-9b85-fbfe27886ab2/serve-healthcheck-canary/0.log"
Jul 14 12:25:17.410890 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:25:17.410861    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-t6fcf/perf-node-gather-daemonset-djh9z"
Jul 14 12:25:17.449524 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:25:17.449494    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dwk5x_3a2ef944-3d8e-41e3-a29f-5c3944611f58/kube-rbac-proxy/0.log"
Jul 14 12:25:17.475695 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:25:17.475668    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dwk5x_3a2ef944-3d8e-41e3-a29f-5c3944611f58/exporter/0.log"
Jul 14 12:25:17.508086 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:25:17.508042    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dwk5x_3a2ef944-3d8e-41e3-a29f-5c3944611f58/extractor/0.log"
Jul 14 12:25:20.283375 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:25:20.283324    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-gl6pr_1540b9c9-a79b-47b4-8183-ea78634934db/manager/0.log"
Jul 14 12:25:20.497696 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:25:20.497660    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-zqj74_ea5ed537-f005-4a9d-8bd2-34dfe38244b1/manager/0.log"
Jul 14 12:25:20.691808 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:25:20.691778    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-ggvrl_dc2f63ce-780c-454a-a5a1-4827c3f08fd8/server/0.log"
Jul 14 12:25:20.777265 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:25:20.777235    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-b9lvb_702307ef-b102-4d06-8246-c81e31a73276/manager/0.log"
Jul 14 12:25:20.989418 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:25:20.989327    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6c944d8cbf-xl5vp_13561ca4-eb2a-4cc6-9abd-3f8ce624cbc9/manager/0.log"
Jul 14 12:25:21.010901 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:25:21.010872    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6c944d8cbf-xl5vp_13561ca4-eb2a-4cc6-9abd-3f8ce624cbc9/copy-manifests/0.log"
Jul 14 12:25:21.019541 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:25:21.019517    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7d55c74c4b-mhfwz_341d716b-2dda-4d65-83d1-a2b4e64ae139/kube-auth-proxy/0.log"
Jul 14 12:25:21.064114 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:25:21.064076    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-r7zl5_2b49d636-77b9-487e-aa76-05a41fe47121/istio-proxy/0.log"
Jul 14 12:25:21.101900 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:25:21.101861    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-r7zl5_2b49d636-77b9-487e-aa76-05a41fe47121/istio-proxy/1.log"
Jul 14 12:25:22.603906 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:25:22.603873    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7d55c74c4b-mhfwz_341d716b-2dda-4d65-83d1-a2b4e64ae139/kube-auth-proxy/0.log"
Jul 14 12:25:22.650044 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:25:22.650014    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-r7zl5_2b49d636-77b9-487e-aa76-05a41fe47121/istio-proxy/0.log"
Jul 14 12:25:22.685808 ip-10-0-136-173 kubenswrapper[2551]: I0714 12:25:22.685773    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-r7zl5_2b49d636-77b9-487e-aa76-05a41fe47121/istio-proxy/1.log"