Jul 19 07:42:12.027561 ip-10-0-136-70 systemd[1]: Starting Kubernetes Kubelet... Jul 19 07:42:14.357665 ip-10-0-136-70 kubenswrapper[2574]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 19 07:42:14.357665 ip-10-0-136-70 kubenswrapper[2574]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 19 07:42:14.357665 ip-10-0-136-70 kubenswrapper[2574]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 19 07:42:14.357665 ip-10-0-136-70 kubenswrapper[2574]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 19 07:42:14.357665 ip-10-0-136-70 kubenswrapper[2574]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 19 07:42:14.358307 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.357674 2574 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 19 07:42:14.365159 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.365138 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 19 07:42:14.365159 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.365155 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 19 07:42:14.365159 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.365159 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 19 07:42:14.365159 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.365162 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 19 07:42:14.365283 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.365166 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 19 07:42:14.365283 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.365169 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 19 07:42:14.365283 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.365172 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 19 07:42:14.365283 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.365174 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 19 07:42:14.365283 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.365177 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 19 07:42:14.365283 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.365186 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 19 07:42:14.365283 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.365190 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 19 07:42:14.365283 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.365193 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 19 07:42:14.365283 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.365196 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 19 07:42:14.365283 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.365198 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 19 07:42:14.365283 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.365201 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 19 07:42:14.365283 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.365204 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 19 07:42:14.365283 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.365206 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 19 07:42:14.365283 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.365209 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 19 07:42:14.365283 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.365212 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 19 07:42:14.365283 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.365214 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 19 07:42:14.365283 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.365217 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 19 07:42:14.365283 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.365219 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 19 07:42:14.365283 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.365222 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 19 07:42:14.365283 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.365225 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 19 07:42:14.365769 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.365227 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 19 07:42:14.365769 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.365229 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 19 07:42:14.365769 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.365232 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 19 07:42:14.365769 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.365235 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 19 07:42:14.365769 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.365237 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 19 07:42:14.365769 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.365240 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 19 07:42:14.365769 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.365243 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 19 07:42:14.365769 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.365247 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 19 07:42:14.365769 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.365250 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 19 07:42:14.365769 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.365253 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 19 07:42:14.365769 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.365256 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 19 07:42:14.365769 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.365259 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 19 07:42:14.365769 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.365262 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 19 07:42:14.365769 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.365265 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 19 07:42:14.365769 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.365268 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 19 07:42:14.365769 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.365270 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 19 07:42:14.365769 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.365273 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 19 07:42:14.365769 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.365275 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 19 07:42:14.365769 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.365278 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 19 07:42:14.365769 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.365280 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 19 07:42:14.366301 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.365283 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 19 07:42:14.366301 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.365286 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 19 07:42:14.366301 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.365289 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 19 07:42:14.366301 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.365291 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 19 07:42:14.366301 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.365294 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 19 07:42:14.366301 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.365297 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 19 07:42:14.366301 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.365299 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 19 07:42:14.366301 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.365302 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 19 07:42:14.366301 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.365304 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 19 07:42:14.366301 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.365307 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 19 07:42:14.366301 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.365309 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 19 07:42:14.366301 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.365312 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 19 07:42:14.366301 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.365314 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 19 07:42:14.366301 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.365317 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 19 07:42:14.366301 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.365320 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 19 07:42:14.366301 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.365322 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 19 07:42:14.366301 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.365327 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 19 07:42:14.366301 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.365331 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 19 07:42:14.366301 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.365334 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 19 07:42:14.367021 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.365336 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 19 07:42:14.367021 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.365339 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 19 07:42:14.367021 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.365342 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 19 07:42:14.367021 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.365345 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 19 07:42:14.367021 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.365348 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 19 07:42:14.367021 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.365351 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 19 07:42:14.367021 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.365355 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 19 07:42:14.367021 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.365358 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 19 07:42:14.367021 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.365361 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 19 07:42:14.367021 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.365364 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 19 07:42:14.367021 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.365366 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 19 07:42:14.367021 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.365369 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 19 07:42:14.367021 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.365371 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 19 07:42:14.367021 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.365374 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 19 07:42:14.367021 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.365377 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 19 07:42:14.367021 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.365379 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 19 07:42:14.367021 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.365382 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 19 07:42:14.367021 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.365384 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 19 07:42:14.367021 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.365387 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 19 07:42:14.367021 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.365389 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 19 07:42:14.367498 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.365393 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 19 07:42:14.367498 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.365397 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 19 07:42:14.367498 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.365400 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 19 07:42:14.367498 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.365403 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 19 07:42:14.367498 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.365406 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 19 07:42:14.367498 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.365409 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 19 07:42:14.367498 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.365411 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 19 07:42:14.367498 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.365414 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 19 07:42:14.367498 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.365416 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 19 07:42:14.367498 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.365419 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 19 07:42:14.367498 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.365422 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 19 07:42:14.367498 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.365424 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 19 07:42:14.367498 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.365427 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 19 07:42:14.367498 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.365430 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 19 07:42:14.367498 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.365433 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 19 07:42:14.367498 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.365435 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 19 07:42:14.369642 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.369625 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 19 07:42:14.369642 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.369640 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 19 07:42:14.369708 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.369644 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 19 07:42:14.369708 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.369648 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 19 07:42:14.369708 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.369651 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 19 07:42:14.369708 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.369654 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 19 07:42:14.369708 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.369657 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 19 07:42:14.369708 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.369660 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 19 07:42:14.369708 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.369663 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 19 07:42:14.369708 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.369665 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 19 07:42:14.369708 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.369668 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 19 07:42:14.369708 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.369670 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 19 07:42:14.369708 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.369673 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 19 07:42:14.369708 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.369676 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 19 07:42:14.369708 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.369678 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 19 07:42:14.369708 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.369681 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 19 07:42:14.369708 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.369683 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 19 07:42:14.369708 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.369686 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 19 07:42:14.369708 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.369688 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 19 07:42:14.369708 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.369691 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 19 07:42:14.369708 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.369693 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 19 07:42:14.369708 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.369696 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 19 07:42:14.370206 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.369698 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 19 07:42:14.370206 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.369701 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 19 07:42:14.370206 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.369703 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 19 07:42:14.370206 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.369705 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 19 07:42:14.370206 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.369708 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 19 07:42:14.370206 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.369710 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 19 07:42:14.370206 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.369713 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 19 07:42:14.370206 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.369716 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 19 07:42:14.370206 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.369719 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 19 07:42:14.370206 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.369722 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 19 07:42:14.370206 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.369729 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 19 07:42:14.370206 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.369732 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 19 07:42:14.370206 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.369735 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 19 07:42:14.370206 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.369738 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 19 07:42:14.370206 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.369740 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 19 07:42:14.370206 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.369743 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 19 07:42:14.370206 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.369745 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 19 07:42:14.370206 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.369749 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 19 07:42:14.370206 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.369753 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 19 07:42:14.370206 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.369755 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 19 07:42:14.370702 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.369758 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 19 07:42:14.370702 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.369760 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 19 07:42:14.370702 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.369763 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 19 07:42:14.370702 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.369766 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 19 07:42:14.370702 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.369768 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 19 07:42:14.370702 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.369770 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 19 07:42:14.370702 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.369773 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 19 07:42:14.370702 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.369777 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 19 07:42:14.370702 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.369781 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 19 07:42:14.370702 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.369783 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 19 07:42:14.370702 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.369786 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 19 07:42:14.370702 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.369789 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 19 07:42:14.370702 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.369791 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 19 07:42:14.370702 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.369795 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 19 07:42:14.370702 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.369798 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 19 07:42:14.370702 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.369801 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 19 07:42:14.370702 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.369804 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 19 07:42:14.370702 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.369808 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 19 07:42:14.370702 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.369810 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 19 07:42:14.371154 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.369813 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 19 07:42:14.371154 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.369816 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 19 07:42:14.371154 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.369818 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 19 07:42:14.371154 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.369821 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 19 07:42:14.371154 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.369825 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 19 07:42:14.371154 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.369828 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 19 07:42:14.371154 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.369831 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 19 07:42:14.371154 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.369834 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 19 07:42:14.371154 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.369837 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 19 07:42:14.371154 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.369839 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 19 07:42:14.371154 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.369842 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 19 07:42:14.371154 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.369844 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 19 07:42:14.371154 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.369847 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 19 07:42:14.371154 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.369849 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 19 07:42:14.371154 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.369852 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 19 07:42:14.371154 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.369854 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 19 07:42:14.371154 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.369857 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 19 07:42:14.371154 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.369859 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 19 07:42:14.371154 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.369862 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 19 07:42:14.371154 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.369865 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 19 07:42:14.371663 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.369867 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 19 07:42:14.371663 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.369870 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 19 07:42:14.371663 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.369873 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 19 07:42:14.371663 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.369876 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 19 07:42:14.371663 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.369879 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 19 07:42:14.371663 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.369882 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 19 07:42:14.371663 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.369885 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 19 07:42:14.371663 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.369888 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 19 07:42:14.371663 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.369891 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 19 07:42:14.371663 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.369893 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 19 07:42:14.371663 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.369897 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 19 07:42:14.371663 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.369899 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 19 07:42:14.371663 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.369902 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 19 07:42:14.371663 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.369905 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 19 07:42:14.371663 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.369907 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 19 07:42:14.371663 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.369910 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 19 07:42:14.371663 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.369912 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 19 07:42:14.371663 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.369915 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 19 07:42:14.371663 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.370331 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 19 07:42:14.372158 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.370336 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 19 07:42:14.372158 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.370341 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 19 07:42:14.372158 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.370344 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 19 07:42:14.372158 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.370347 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 19 07:42:14.372158 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.370350 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 19 07:42:14.372158 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.370353 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 19 07:42:14.372158 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.370356 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 19 07:42:14.372158 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.370359 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 19 07:42:14.372158 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.370361 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 19 07:42:14.372158 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.370364 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 19 07:42:14.372158 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.370367 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 19 07:42:14.372158 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.370369 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 19 07:42:14.372158 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.370372 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 19 07:42:14.372158 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.370374 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 19 07:42:14.372158 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.370377 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 19 07:42:14.372158 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.370380 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 19 07:42:14.372158 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.370383 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 19 07:42:14.372158 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.370385 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 19 07:42:14.372158 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.370388 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 19 07:42:14.372622 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.370391 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 19 07:42:14.372622 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.370394 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 19 07:42:14.372622 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.370396 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 19 07:42:14.372622 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.370399 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 19 07:42:14.372622 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.370401 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 19 07:42:14.372622 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.370404 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 19 07:42:14.372622 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.370406 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 19 07:42:14.372622 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.370409 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 19 07:42:14.372622 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.370412 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 19 07:42:14.372622 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.370414 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 19 07:42:14.372622 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.370417 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 19 07:42:14.372622 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.370419 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 19 07:42:14.372622 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.370422 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 19 07:42:14.372622 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.370425 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 19 07:42:14.372622 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.370427 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 19 07:42:14.372622 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.370430 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 19 07:42:14.372622 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.370433 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 19 07:42:14.372622 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.370435 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 19 07:42:14.372622 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.370438 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 19 07:42:14.372622 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.370441 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 19 07:42:14.373110 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.370443 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 19 07:42:14.373110 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.370445 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 19 07:42:14.373110 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.370449 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 19 07:42:14.373110 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.370451 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 19 07:42:14.373110 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.370454 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 19 07:42:14.373110 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.370456 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 19 07:42:14.373110 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.370459 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 19 07:42:14.373110 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.370461 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 19 07:42:14.373110 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.370464 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 19 07:42:14.373110 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.370467 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 19 07:42:14.373110 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.370469 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 19 07:42:14.373110 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.370475 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 19 07:42:14.373110 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.370477 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 19 07:42:14.373110 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.370480 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 19 07:42:14.373110 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.370483 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 19 07:42:14.373110 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.370485 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 19 07:42:14.373110 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.370488 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 19 07:42:14.373110 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.370490 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 19 07:42:14.373110 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.370493 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 19 07:42:14.373110 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.370495 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 19 07:42:14.373610 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.370498 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 19 07:42:14.373610 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.370500 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 19 07:42:14.373610 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.370504 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 19 07:42:14.373610 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.370508 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 19 07:42:14.373610 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.370510 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 19 07:42:14.373610 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.370513 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 19 07:42:14.373610 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.370529 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 19 07:42:14.373610 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.370532 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 19 07:42:14.373610 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.370536 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 19 07:42:14.373610 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.370538 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 19 07:42:14.373610 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.370541 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 19 07:42:14.373610 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.370544 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 19 07:42:14.373610 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.370547 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 19 07:42:14.373610 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.370549 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 19 07:42:14.373610 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.370552 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 19 07:42:14.373610 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.370555 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 19 07:42:14.373610 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.370558 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 19 07:42:14.373610 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.370561 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 19 07:42:14.373610 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.370563 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 19 07:42:14.374063 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.370567 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 19 07:42:14.374063 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.370569 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 19 07:42:14.374063 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.370572 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 19 07:42:14.374063 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.370575 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 19 07:42:14.374063 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.370579 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 19 07:42:14.374063 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.370582 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 19 07:42:14.374063 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.370585 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 19 07:42:14.374063 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.370587 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 19 07:42:14.374063 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.370589 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 19 07:42:14.374063 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.370592 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 19 07:42:14.374063 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.370595 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 19 07:42:14.374063 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.370597 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 19 07:42:14.374063 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.370600 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 19 07:42:14.374063 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.370602 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 19 07:42:14.374063 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.370606 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 19 07:42:14.374063 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.370610 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 19 07:42:14.374063 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.370614 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 19 07:42:14.374063 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.370616 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 19 07:42:14.374063 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.370619 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 19 07:42:14.374063 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.370622 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 19 07:42:14.374548 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.370690 2574 flags.go:64] FLAG: --address="0.0.0.0" Jul 19 07:42:14.374548 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.370698 2574 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 19 07:42:14.374548 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.370704 2574 flags.go:64] FLAG: --anonymous-auth="true" Jul 19 07:42:14.374548 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.370708 2574 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 19 07:42:14.374548 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.370712 2574 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 19 07:42:14.374548 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.370716 2574 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 19 07:42:14.374548 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.370720 2574 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 19 07:42:14.374548 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.370725 2574 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 19 07:42:14.374548 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.370728 2574 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 19 07:42:14.374548 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.370731 2574 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 19 07:42:14.374548 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.370735 2574 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 19 07:42:14.374548 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.370738 2574 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 19 07:42:14.374548 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.370741 2574 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 19 07:42:14.374548 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.370744 2574 flags.go:64] FLAG: --cgroup-root="" Jul 19 07:42:14.374548 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.370747 2574 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 19 07:42:14.374548 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.370750 2574 flags.go:64] FLAG: --client-ca-file="" Jul 19 07:42:14.374548 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.370755 2574 flags.go:64] FLAG: --cloud-provider="external" Jul 19 07:42:14.374548 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.370758 2574 flags.go:64] FLAG: --cluster-dns="[]" Jul 19 07:42:14.374548 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.370768 2574 flags.go:64] FLAG: --cluster-domain="" Jul 19 07:42:14.374548 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.370771 2574 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 19 07:42:14.374548 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.370774 2574 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 19 07:42:14.374548 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.370778 2574 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 19 07:42:14.374548 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.370781 2574 flags.go:64] FLAG: --container-log-max-files="5" Jul 19 07:42:14.374548 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.370785 2574 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 19 07:42:14.375128 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.370787 2574 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 19 07:42:14.375128 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.370790 2574 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 19 07:42:14.375128 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.370794 2574 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 19 07:42:14.375128 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.370797 2574 flags.go:64] FLAG: --contention-profiling="false" Jul 19 07:42:14.375128 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.370800 2574 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 19 07:42:14.375128 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.370803 2574 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 19 07:42:14.375128 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.370806 2574 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 19 07:42:14.375128 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.370809 2574 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 19 07:42:14.375128 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.370813 2574 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 19 07:42:14.375128 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.370816 2574 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 19 07:42:14.375128 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.370819 2574 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 19 07:42:14.375128 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.370822 2574 flags.go:64] FLAG: --enable-load-reader="false" Jul 19 07:42:14.375128 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.370825 2574 flags.go:64] FLAG: --enable-server="true" Jul 19 07:42:14.375128 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.370828 2574 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 19 07:42:14.375128 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.370832 2574 flags.go:64] FLAG: --event-burst="100" Jul 19 07:42:14.375128 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.370835 2574 flags.go:64] FLAG: --event-qps="50" Jul 19 07:42:14.375128 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.370838 2574 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 19 07:42:14.375128 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.370841 2574 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 19 07:42:14.375128 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.370844 2574 flags.go:64] FLAG: --eviction-hard="" Jul 19 07:42:14.375128 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.370847 2574 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 19 07:42:14.375128 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.370850 2574 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 19 07:42:14.375128 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.370853 2574 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 19 07:42:14.375128 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.370856 2574 flags.go:64] FLAG: --eviction-soft="" Jul 19 07:42:14.375128 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.370859 2574 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 19 07:42:14.375128 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.370863 2574 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 19 07:42:14.375785 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.370869 2574 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 19 07:42:14.375785 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.370872 2574 flags.go:64] FLAG: --experimental-mounter-path="" Jul 19 07:42:14.375785 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.370875 2574 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 19 07:42:14.375785 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.370878 2574 flags.go:64] FLAG: --fail-swap-on="true" Jul 19 07:42:14.375785 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.370881 2574 flags.go:64] FLAG: --feature-gates="" Jul 19 07:42:14.375785 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.370884 2574 flags.go:64] FLAG: --file-check-frequency="20s" Jul 19 07:42:14.375785 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.370887 2574 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 19 07:42:14.375785 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.370890 2574 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 19 07:42:14.375785 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.370893 2574 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 19 07:42:14.375785 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.370896 2574 flags.go:64] FLAG: --healthz-port="10248" Jul 19 07:42:14.375785 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.370899 2574 flags.go:64] FLAG: --help="false" Jul 19 07:42:14.375785 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.370902 2574 flags.go:64] FLAG: --hostname-override="ip-10-0-136-70.ec2.internal" Jul 19 07:42:14.375785 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.370905 2574 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 19 07:42:14.375785 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.370908 2574 flags.go:64] FLAG: --http-check-frequency="20s" Jul 19 07:42:14.375785 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.370911 2574 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 19 07:42:14.375785 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.370914 2574 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 19 07:42:14.375785 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.370918 2574 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 19 07:42:14.375785 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.370921 2574 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 19 07:42:14.375785 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.370924 2574 flags.go:64] FLAG: --image-service-endpoint="" Jul 19 07:42:14.375785 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.370927 2574 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 19 07:42:14.375785 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.370930 2574 flags.go:64] FLAG: --kube-api-burst="100" Jul 19 07:42:14.375785 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.370932 2574 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 19 07:42:14.375785 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.370935 2574 flags.go:64] FLAG: --kube-api-qps="50" Jul 19 07:42:14.375785 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.370938 2574 flags.go:64] FLAG: --kube-reserved="" Jul 19 07:42:14.376371 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.370941 2574 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 19 07:42:14.376371 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.370944 2574 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 19 07:42:14.376371 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.370947 2574 flags.go:64] FLAG: --kubelet-cgroups="" Jul 19 07:42:14.376371 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.370949 2574 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 19 07:42:14.376371 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.370952 2574 flags.go:64] FLAG: --lock-file="" Jul 19 07:42:14.376371 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.370955 2574 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 19 07:42:14.376371 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.370958 2574 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 19 07:42:14.376371 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.370962 2574 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 19 07:42:14.376371 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.370973 2574 flags.go:64] FLAG: --log-json-split-stream="false" Jul 19 07:42:14.376371 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.370976 2574 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 19 07:42:14.376371 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.370979 2574 flags.go:64] FLAG: --log-text-split-stream="false" Jul 19 07:42:14.376371 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.370982 2574 flags.go:64] FLAG: --logging-format="text" Jul 19 07:42:14.376371 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.370985 2574 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 19 07:42:14.376371 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.370989 2574 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 19 07:42:14.376371 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.370991 2574 flags.go:64] FLAG: --manifest-url="" Jul 19 07:42:14.376371 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.370994 2574 flags.go:64] FLAG: --manifest-url-header="" Jul 19 07:42:14.376371 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.370999 2574 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 19 07:42:14.376371 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.371001 2574 flags.go:64] FLAG: --max-open-files="1000000" Jul 19 07:42:14.376371 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.371005 2574 flags.go:64] FLAG: --max-pods="110" Jul 19 07:42:14.376371 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.371008 2574 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 19 07:42:14.376371 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.371011 2574 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 19 07:42:14.376371 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.371014 2574 flags.go:64] FLAG: --memory-manager-policy="None" Jul 19 07:42:14.376371 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.371017 2574 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 19 07:42:14.376371 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.371020 2574 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 19 07:42:14.376371 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.371023 2574 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 19 07:42:14.376986 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.371026 2574 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 19 07:42:14.376986 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.371033 2574 flags.go:64] FLAG: --node-status-max-images="50" Jul 19 07:42:14.376986 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.371037 2574 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 19 07:42:14.376986 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.371040 2574 flags.go:64] FLAG: --oom-score-adj="-999" Jul 19 07:42:14.376986 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.371043 2574 flags.go:64] FLAG: --pod-cidr="" Jul 19 07:42:14.376986 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.371045 2574 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 19 07:42:14.376986 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.371051 2574 flags.go:64] FLAG: --pod-manifest-path="" Jul 19 07:42:14.376986 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.371054 2574 flags.go:64] FLAG: --pod-max-pids="-1" Jul 19 07:42:14.376986 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.371057 2574 flags.go:64] FLAG: --pods-per-core="0" Jul 19 07:42:14.376986 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.371059 2574 flags.go:64] FLAG: --port="10250" Jul 19 07:42:14.376986 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.371062 2574 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 19 07:42:14.376986 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.371065 2574 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-03da91bc8ec9b5749" Jul 19 07:42:14.376986 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.371069 2574 flags.go:64] FLAG: --qos-reserved="" Jul 19 07:42:14.376986 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.371072 2574 flags.go:64] FLAG: --read-only-port="10255" Jul 19 07:42:14.376986 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.371075 2574 flags.go:64] FLAG: --register-node="true" Jul 19 07:42:14.376986 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.371082 2574 flags.go:64] FLAG: --register-with-taints="" Jul 19 07:42:14.376986 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.371086 2574 flags.go:64] FLAG: --registry-burst="10" Jul 19 07:42:14.376986 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.371089 2574 flags.go:64] FLAG: --registry-qps="5" Jul 19 07:42:14.376986 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.371092 2574 flags.go:64] FLAG: --reserved-cpus="" Jul 19 07:42:14.376986 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.371095 2574 flags.go:64] FLAG: --reserved-memory="" Jul 19 07:42:14.376986 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.371099 2574 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 19 07:42:14.376986 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.371102 2574 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 19 07:42:14.376986 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.371105 2574 flags.go:64] FLAG: --rotate-certificates="false" Jul 19 07:42:14.376986 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.371107 2574 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 19 07:42:14.376986 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.371110 2574 flags.go:64] FLAG: --runonce="false" Jul 19 07:42:14.377637 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.371113 2574 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 19 07:42:14.377637 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.371116 2574 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 19 07:42:14.377637 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.371119 2574 flags.go:64] FLAG: --seccomp-default="false" Jul 19 07:42:14.377637 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.371122 2574 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 19 07:42:14.377637 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.371124 2574 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 19 07:42:14.377637 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.371127 2574 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 19 07:42:14.377637 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.371130 2574 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 19 07:42:14.377637 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.371133 2574 flags.go:64] FLAG: --storage-driver-password="root" Jul 19 07:42:14.377637 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.371136 2574 flags.go:64] FLAG: --storage-driver-secure="false" Jul 19 07:42:14.377637 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.371139 2574 flags.go:64] FLAG: --storage-driver-table="stats" Jul 19 07:42:14.377637 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.371142 2574 flags.go:64] FLAG: --storage-driver-user="root" Jul 19 07:42:14.377637 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.371145 2574 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 19 07:42:14.377637 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.371148 2574 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 19 07:42:14.377637 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.371151 2574 flags.go:64] FLAG: --system-cgroups="" Jul 19 07:42:14.377637 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.371153 2574 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 19 07:42:14.377637 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.371159 2574 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 19 07:42:14.377637 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.371161 2574 flags.go:64] FLAG: --tls-cert-file="" Jul 19 07:42:14.377637 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.371164 2574 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 19 07:42:14.377637 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.371169 2574 flags.go:64] FLAG: --tls-min-version="" Jul 19 07:42:14.377637 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.371172 2574 flags.go:64] FLAG: --tls-private-key-file="" Jul 19 07:42:14.377637 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.371174 2574 flags.go:64] FLAG: --topology-manager-policy="none" Jul 19 07:42:14.377637 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.371177 2574 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 19 07:42:14.377637 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.371183 2574 flags.go:64] FLAG: --topology-manager-scope="container" Jul 19 07:42:14.377637 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.371185 2574 flags.go:64] FLAG: --v="2" Jul 19 07:42:14.377637 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.371189 2574 flags.go:64] FLAG: --version="false" Jul 19 07:42:14.378244 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.371194 2574 flags.go:64] FLAG: --vmodule="" Jul 19 07:42:14.378244 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.371198 2574 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 19 07:42:14.378244 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.371201 2574 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 19 07:42:14.378244 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.371332 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 19 07:42:14.378244 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.371337 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 19 07:42:14.378244 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.371340 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 19 07:42:14.378244 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.371343 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 19 07:42:14.378244 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.371345 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 19 07:42:14.378244 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.371348 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 19 07:42:14.378244 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.371351 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 19 07:42:14.378244 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.371355 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 19 07:42:14.378244 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.371359 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 19 07:42:14.378244 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.371362 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 19 07:42:14.378244 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.371364 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 19 07:42:14.378244 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.371367 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 19 07:42:14.378244 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.371369 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 19 07:42:14.378244 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.371371 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 19 07:42:14.378244 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.371374 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 19 07:42:14.378244 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.371378 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 19 07:42:14.378717 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.371382 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 19 07:42:14.378717 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.371385 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 19 07:42:14.378717 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.371388 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 19 07:42:14.378717 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.371391 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 19 07:42:14.378717 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.371394 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 19 07:42:14.378717 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.371397 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 19 07:42:14.378717 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.371401 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 19 07:42:14.378717 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.371404 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 19 07:42:14.378717 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.371406 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 19 07:42:14.378717 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.371409 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 19 07:42:14.378717 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.371414 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 19 07:42:14.378717 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.371417 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 19 07:42:14.378717 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.371420 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 19 07:42:14.378717 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.371423 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 19 07:42:14.378717 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.371426 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 19 07:42:14.378717 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.371428 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 19 07:42:14.378717 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.371431 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 19 07:42:14.378717 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.371434 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 19 07:42:14.378717 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.371441 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 19 07:42:14.379190 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.371444 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 19 07:42:14.379190 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.371446 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 19 07:42:14.379190 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.371449 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 19 07:42:14.379190 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.371452 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 19 07:42:14.379190 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.371455 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 19 07:42:14.379190 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.371457 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 19 07:42:14.379190 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.371460 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 19 07:42:14.379190 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.371462 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 19 07:42:14.379190 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.371464 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 19 07:42:14.379190 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.371467 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 19 07:42:14.379190 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.371469 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 19 07:42:14.379190 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.371472 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 19 07:42:14.379190 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.371475 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 19 07:42:14.379190 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.371477 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 19 07:42:14.379190 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.371480 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 19 07:42:14.379190 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.371482 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 19 07:42:14.379190 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.371485 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 19 07:42:14.379190 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.371487 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 19 07:42:14.379190 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.371490 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 19 07:42:14.379190 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.371492 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 19 07:42:14.379735 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.371495 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 19 07:42:14.379735 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.371497 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 19 07:42:14.379735 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.371500 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 19 07:42:14.379735 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.371507 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 19 07:42:14.379735 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.371509 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 19 07:42:14.379735 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.371512 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 19 07:42:14.379735 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.371527 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 19 07:42:14.379735 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.371530 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 19 07:42:14.379735 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.371534 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 19 07:42:14.379735 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.371537 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 19 07:42:14.379735 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.371539 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 19 07:42:14.379735 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.371542 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 19 07:42:14.379735 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.371545 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 19 07:42:14.379735 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.371547 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 19 07:42:14.379735 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.371550 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 19 07:42:14.379735 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.371553 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 19 07:42:14.379735 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.371555 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 19 07:42:14.379735 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.371558 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 19 07:42:14.379735 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.371561 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 19 07:42:14.379735 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.371563 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 19 07:42:14.380220 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.371566 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 19 07:42:14.380220 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.371568 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 19 07:42:14.380220 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.371571 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 19 07:42:14.380220 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.371574 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 19 07:42:14.380220 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.371576 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 19 07:42:14.380220 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.371579 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 19 07:42:14.380220 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.371582 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 19 07:42:14.380220 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.371585 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 19 07:42:14.380220 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.371589 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 19 07:42:14.380220 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.371592 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 19 07:42:14.380220 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.371595 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 19 07:42:14.380220 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.371598 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 19 07:42:14.380220 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.371600 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 19 07:42:14.380220 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.371603 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 19 07:42:14.380220 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.371605 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 19 07:42:14.380220 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.371611 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 19 07:42:14.380220 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.371614 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 19 07:42:14.380220 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.371616 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 19 07:42:14.380220 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.371619 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 19 07:42:14.380220 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.371622 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 19 07:42:14.380842 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.371625 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 19 07:42:14.380842 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.371631 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 19 07:42:14.380842 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.371634 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 19 07:42:14.380842 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.371637 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 19 07:42:14.380842 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.371643 2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 19 07:42:14.380842 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.379077 2574 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 19 07:42:14.380842 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.379093 2574 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 19 07:42:14.380842 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.380016 2574 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 19 07:42:14.380842 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.380027 2574 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 19 07:42:14.380842 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380087 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 19 07:42:14.380842 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380092 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 19 07:42:14.380842 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380096 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 19 07:42:14.380842 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380100 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 19 07:42:14.380842 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380103 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 19 07:42:14.380842 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380106 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 19 07:42:14.381430 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380110 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 19 07:42:14.381430 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380114 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 19 07:42:14.381430 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380117 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 19 07:42:14.381430 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380120 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 19 07:42:14.381430 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380123 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 19 07:42:14.381430 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380126 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 19 07:42:14.381430 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380128 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 19 07:42:14.381430 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380131 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 19 07:42:14.381430 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380134 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 19 07:42:14.381430 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380136 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 19 07:42:14.381430 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380139 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 19 07:42:14.381430 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380142 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 19 07:42:14.381430 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380145 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 19 07:42:14.381430 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380147 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 19 07:42:14.381430 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380150 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 19 07:42:14.381430 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380152 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 19 07:42:14.381430 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380155 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 19 07:42:14.381430 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380158 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 19 07:42:14.381430 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380161 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 19 07:42:14.381430 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380164 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 19 07:42:14.381997 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380167 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 19 07:42:14.381997 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380170 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 19 07:42:14.381997 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380172 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 19 07:42:14.381997 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380176 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 19 07:42:14.381997 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380179 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 19 07:42:14.381997 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380181 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 19 07:42:14.381997 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380184 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 19 07:42:14.381997 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380187 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 19 07:42:14.381997 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380190 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 19 07:42:14.381997 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380192 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 19 07:42:14.381997 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380194 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 19 07:42:14.381997 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380197 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 19 07:42:14.381997 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380200 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 19 07:42:14.381997 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380202 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 19 07:42:14.381997 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380205 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 19 07:42:14.381997 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380207 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 19 07:42:14.381997 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380210 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 19 07:42:14.381997 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380212 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 19 07:42:14.381997 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380215 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 19 07:42:14.381997 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380218 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 19 07:42:14.382814 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380220 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 19 07:42:14.382814 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380222 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 19 07:42:14.382814 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380225 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 19 07:42:14.382814 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380227 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 19 07:42:14.382814 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380230 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 19 07:42:14.382814 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380233 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 19 07:42:14.382814 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380236 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 19 07:42:14.382814 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380238 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 19 07:42:14.382814 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380241 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 19 07:42:14.382814 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380243 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 19 07:42:14.382814 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380246 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 19 07:42:14.382814 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380249 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 19 07:42:14.382814 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380253 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 19 07:42:14.382814 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380256 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 19 07:42:14.382814 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380259 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 19 07:42:14.382814 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380263 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 19 07:42:14.382814 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380266 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 19 07:42:14.382814 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380269 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 19 07:42:14.382814 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380272 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 19 07:42:14.383648 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380275 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 19 07:42:14.383648 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380279 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 19 07:42:14.383648 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380282 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 19 07:42:14.383648 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380285 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 19 07:42:14.383648 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380288 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 19 07:42:14.383648 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380291 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 19 07:42:14.383648 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380293 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 19 07:42:14.383648 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380296 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 19 07:42:14.383648 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380298 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 19 07:42:14.383648 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380301 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 19 07:42:14.383648 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380304 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 19 07:42:14.383648 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380307 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 19 07:42:14.383648 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380309 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 19 07:42:14.383648 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380312 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 19 07:42:14.383648 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380314 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 19 07:42:14.383648 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380317 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 19 07:42:14.383648 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380319 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 19 07:42:14.383648 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380322 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 19 07:42:14.383648 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380324 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 19 07:42:14.384163 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380327 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 19 07:42:14.384163 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380329 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 19 07:42:14.384163 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380332 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 19 07:42:14.384163 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380335 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 19 07:42:14.384163 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380337 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 19 07:42:14.384163 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380340 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 19 07:42:14.384163 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380343 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 19 07:42:14.384163 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380346 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 19 07:42:14.384163 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380348 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 19 07:42:14.384163 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380352 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 19 07:42:14.384163 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380354 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 19 07:42:14.384163 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380357 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 19 07:42:14.384163 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380360 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 19 07:42:14.384163 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380362 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 19 07:42:14.384163 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380365 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 19 07:42:14.384647 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.380370 2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 19 07:42:14.384647 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380480 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 19 07:42:14.384647 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380484 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 19 07:42:14.384647 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380487 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 19 07:42:14.384647 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380490 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 19 07:42:14.384647 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380493 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 19 07:42:14.384647 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380495 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 19 07:42:14.384647 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380498 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 19 07:42:14.384647 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380501 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 19 07:42:14.384647 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380504 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 19 07:42:14.384647 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380506 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 19 07:42:14.384647 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380509 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 19 07:42:14.384647 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380511 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 19 07:42:14.384647 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380527 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 19 07:42:14.384647 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380531 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 19 07:42:14.385035 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380534 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 19 07:42:14.385035 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380537 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 19 07:42:14.385035 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380574 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 19 07:42:14.385035 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380584 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 19 07:42:14.385035 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380589 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 19 07:42:14.385035 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380594 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 19 07:42:14.385035 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380601 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 19 07:42:14.385035 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380606 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 19 07:42:14.385035 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380610 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 19 07:42:14.385035 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380617 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 19 07:42:14.385035 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380622 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 19 07:42:14.385035 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380626 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 19 07:42:14.385035 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380631 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 19 07:42:14.385035 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380635 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 19 07:42:14.385035 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380639 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 19 07:42:14.385035 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380644 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 19 07:42:14.385035 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380648 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 19 07:42:14.385035 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380653 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 19 07:42:14.385035 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380657 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 19 07:42:14.385483 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380662 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 19 07:42:14.385483 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380666 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 19 07:42:14.385483 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380674 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 19 07:42:14.385483 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380680 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 19 07:42:14.385483 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380685 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 19 07:42:14.385483 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380690 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 19 07:42:14.385483 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380695 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 19 07:42:14.385483 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380699 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 19 07:42:14.385483 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380704 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 19 07:42:14.385483 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380708 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 19 07:42:14.385483 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380712 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 19 07:42:14.385483 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380720 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 19 07:42:14.385483 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380726 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 19 07:42:14.385483 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380731 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 19 07:42:14.385483 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380736 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 19 07:42:14.385483 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380741 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 19 07:42:14.385483 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380745 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 19 07:42:14.385483 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380750 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 19 07:42:14.385483 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380754 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 19 07:42:14.385483 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380759 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 19 07:42:14.386035 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380765 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 19 07:42:14.386035 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380769 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 19 07:42:14.386035 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380773 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 19 07:42:14.386035 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380779 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 19 07:42:14.386035 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380784 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 19 07:42:14.386035 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380788 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 19 07:42:14.386035 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380792 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 19 07:42:14.386035 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380796 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 19 07:42:14.386035 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380800 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 19 07:42:14.386035 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380805 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 19 07:42:14.386035 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380809 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 19 07:42:14.386035 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380814 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 19 07:42:14.386035 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380818 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 19 07:42:14.386035 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380822 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 19 07:42:14.386035 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380827 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 19 07:42:14.386035 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380831 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 19 07:42:14.386035 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380837 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 19 07:42:14.386035 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380841 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 19 07:42:14.386035 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380845 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 19 07:42:14.386035 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380849 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 19 07:42:14.386512 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380853 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 19 07:42:14.386512 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380857 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 19 07:42:14.386512 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380861 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 19 07:42:14.386512 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380865 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 19 07:42:14.386512 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380871 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 19 07:42:14.386512 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380875 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 19 07:42:14.386512 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380880 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 19 07:42:14.386512 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380885 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 19 07:42:14.386512 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380890 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 19 07:42:14.386512 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380894 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 19 07:42:14.386512 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380899 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 19 07:42:14.386512 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380903 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 19 07:42:14.386512 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380907 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 19 07:42:14.386512 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380912 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 19 07:42:14.386512 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380916 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 19 07:42:14.386512 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380920 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 19 07:42:14.386512 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380926 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 19 07:42:14.386512 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380932 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 19 07:42:14.386512 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380939 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 19 07:42:14.387048 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380943 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 19 07:42:14.387048 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380948 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 19 07:42:14.387048 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380952 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 19 07:42:14.387048 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380957 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 19 07:42:14.387048 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380961 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 19 07:42:14.387048 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380966 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 19 07:42:14.387048 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:14.380970 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 19 07:42:14.387048 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.380979 2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 19 07:42:14.387048 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.381155 2574 server.go:962] "Client rotation is on, will bootstrap in background" Jul 19 07:42:14.387048 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.384182 2574 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 19 07:42:14.387048 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.385932 2574 server.go:1019] "Starting client certificate rotation" Jul 19 07:42:14.387048 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.386047 2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 19 07:42:14.387048 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.386084 2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 19 07:42:14.388848 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.388829 2574 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 19 07:42:14.390066 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.390036 2574 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 19 07:42:14.400728 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.399734 2574 log.go:25] "Validated CRI v1 runtime API" Jul 19 07:42:14.404843 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.404826 2574 log.go:25] "Validated CRI v1 image API" Jul 19 07:42:14.406600 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.406583 2574 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 19 07:42:14.410120 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.410100 2574 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 90cd2862-65f6-408a-8aaa-16f76b005422:/dev/nvme0n1p4 e2122fb3-b1b0-46bd-83fa-ac4e28f7ff90:/dev/nvme0n1p3] Jul 19 07:42:14.410194 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.410118 2574 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 19 07:42:14.415545 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.415400 2574 manager.go:216] Machine: {Timestamp:2026-07-19 07:42:14.4139886 +0000 UTC m=+0.768298192 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100233 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec28644cae5716bf40d8b4fa1c159507 SystemUUID:ec28644c-ae57-16bf-40d8-b4fa1c159507 BootID:5aadc063-53be-45bd-8f47-f7c789aabfd3 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:a0:bb:ff:13:2b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:a0:bb:ff:13:2b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:6a:7d:2c:b7:ed:48 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 19 07:42:14.415545 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.415542 2574 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 19 07:42:14.415718 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.415702 2574 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 19 07:42:14.417252 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.417236 2574 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 19 07:42:14.418574 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.418546 2574 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 19 07:42:14.418747 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.418578 2574 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-70.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 19 07:42:14.419473 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.419462 2574 topology_manager.go:138] "Creating topology manager with none policy" Jul 19 07:42:14.419503 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.419476 2574 container_manager_linux.go:309] "Creating device plugin manager" Jul 19 07:42:14.419503 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.419484 2574 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 19 07:42:14.419573 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.419502 2574 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 19 07:42:14.421467 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.421456 2574 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 19 07:42:14.426187 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.426176 2574 state_mem.go:36] "Initialized new in-memory state store" Jul 19 07:42:14.426318 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.426309 2574 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 19 07:42:14.426349 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.426340 2574 kubelet.go:486] "Attempting to sync node with API server" Jul 19 07:42:14.426376 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.426353 2574 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 19 07:42:14.426376 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.426368 2574 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 19 07:42:14.426430 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.426377 2574 kubelet.go:398] "Adding apiserver pod source" Jul 19 07:42:14.426430 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.426386 2574 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 19 07:42:14.426807 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.426795 2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 19 07:42:14.426862 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.426815 2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 19 07:42:14.429199 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.429177 2574 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 19 07:42:14.431067 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.431054 2574 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 19 07:42:14.431106 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.431084 2574 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 19 07:42:14.431858 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.431848 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 19 07:42:14.431890 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.431863 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 19 07:42:14.431890 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.431869 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 19 07:42:14.431890 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.431874 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 19 07:42:14.431890 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.431882 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 19 07:42:14.431890 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.431887 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 19 07:42:14.432031 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.431938 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 19 07:42:14.432031 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.431945 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 19 07:42:14.432031 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.431962 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 19 07:42:14.432031 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.431969 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 19 07:42:14.432031 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.431977 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 19 07:42:14.432031 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.431985 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 19 07:42:14.433750 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.433736 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 19 07:42:14.433790 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.433754 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 19 07:42:14.437428 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.437413 2574 server.go:1268] "Started kubelet" Jul 19 07:42:14.437534 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.437469 2574 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 19 07:42:14.437603 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.437552 2574 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 19 07:42:14.437705 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.437661 2574 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 19 07:42:14.438488 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:14.438415 2574 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 19 07:42:14.438502 ip-10-0-136-70 systemd[1]: Started Kubernetes Kubelet. Jul 19 07:42:14.438887 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.438680 2574 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-70.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 19 07:42:14.439008 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:14.438862 2574 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-70.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 19 07:42:14.439706 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.439667 2574 server.go:310] "Adding debug handlers to kubelet server" Jul 19 07:42:14.442372 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.442319 2574 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 19 07:42:14.445289 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:14.445261 2574 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 19 07:42:14.447755 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:14.446676 2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-70.ec2.internal.18c3a1084facbfda default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-70.ec2.internal,UID:ip-10-0-136-70.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-70.ec2.internal,},FirstTimestamp:2026-07-19 07:42:14.437380058 +0000 UTC m=+0.791689651,LastTimestamp:2026-07-19 07:42:14.437380058 +0000 UTC m=+0.791689651,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-70.ec2.internal,}" Jul 19 07:42:14.451480 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.451466 2574 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 19 07:42:14.451555 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.451473 2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 19 07:42:14.451615 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:14.451561 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-70.ec2.internal\" not found" Jul 19 07:42:14.452229 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.452207 2574 factory.go:55] Registering systemd factory Jul 19 07:42:14.452229 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.452226 2574 factory.go:223] Registration of the systemd container factory successfully Jul 19 07:42:14.452366 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.452251 2574 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 19 07:42:14.452366 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.452252 2574 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 19 07:42:14.452366 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.452290 2574 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 19 07:42:14.452504 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.452374 2574 reconstruct.go:97] "Volume reconstruction finished" Jul 19 07:42:14.452504 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.452383 2574 reconciler.go:29] "Reconciler: start to sync state" Jul 19 07:42:14.453053 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.452796 2574 factory.go:153] Registering CRI-O factory Jul 19 07:42:14.453053 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.452811 2574 factory.go:223] Registration of the crio container factory successfully Jul 19 07:42:14.453053 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.452858 2574 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 19 07:42:14.453053 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.452883 2574 factory.go:103] Registering Raw factory Jul 19 07:42:14.453053 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.452903 2574 manager.go:1196] Started watching for new ooms in manager Jul 19 07:42:14.453455 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.453330 2574 manager.go:349] Starting recovery of all containers Jul 19 07:42:14.454454 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:14.454425 2574 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 19 07:42:14.454561 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:14.454540 2574 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-70.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 19 07:42:14.463438 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.463417 2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-n8frc" Jul 19 07:42:14.465475 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.465459 2574 manager.go:354] Recovery completed Jul 19 07:42:14.470026 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.470013 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 19 07:42:14.470715 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.470700 2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-n8frc" Jul 19 07:42:14.472167 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.472151 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-70.ec2.internal" event="NodeHasSufficientMemory" Jul 19 07:42:14.472225 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.472181 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-70.ec2.internal" event="NodeHasNoDiskPressure" Jul 19 07:42:14.472225 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.472194 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-70.ec2.internal" event="NodeHasSufficientPID" Jul 19 07:42:14.472754 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.472742 2574 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 19 07:42:14.472754 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.472750 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 19 07:42:14.472861 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.472752 2574 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 19 07:42:14.472861 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.472796 2574 state_mem.go:36] "Initialized new in-memory state store" Jul 19 07:42:14.473514 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.473502 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-70.ec2.internal" event="NodeHasSufficientMemory" Jul 19 07:42:14.473569 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.473563 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-70.ec2.internal" event="NodeHasNoDiskPressure" Jul 19 07:42:14.473607 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.473574 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-70.ec2.internal" event="NodeHasSufficientPID" Jul 19 07:42:14.473920 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:14.473844 2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-70.ec2.internal.18c3a10851bf8e93 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-70.ec2.internal,UID:ip-10-0-136-70.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-136-70.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-136-70.ec2.internal,},FirstTimestamp:2026-07-19 07:42:14.472167059 +0000 UTC m=+0.826476654,LastTimestamp:2026-07-19 07:42:14.472167059 +0000 UTC m=+0.826476654,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-70.ec2.internal,}" Jul 19 07:42:14.475009 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.474995 2574 policy_none.go:49] "None policy: Start" Jul 19 07:42:14.475073 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.475015 2574 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 19 07:42:14.475073 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.475029 2574 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 19 07:42:14.477314 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.477302 2574 policy_none.go:47] "Start" Jul 19 07:42:14.512396 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.512184 2574 manager.go:337] "Starting Device Plugin manager" Jul 19 07:42:14.528324 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:14.512406 2574 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 19 07:42:14.528324 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.512422 2574 server.go:87] "Starting device plugin registration server" Jul 19 07:42:14.528324 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.512779 2574 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 19 07:42:14.528324 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.512792 2574 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 19 07:42:14.528324 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.512904 2574 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 19 07:42:14.528324 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.512987 2574 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 19 07:42:14.528324 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.512996 2574 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 19 07:42:14.528324 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:14.513617 2574 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 19 07:42:14.528324 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:14.513661 2574 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-70.ec2.internal\" not found" Jul 19 07:42:14.550770 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.550726 2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 19 07:42:14.551906 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.551887 2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 19 07:42:14.552048 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.551923 2574 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 19 07:42:14.552048 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.551947 2574 kubelet.go:2448] "Starting kubelet main sync loop" Jul 19 07:42:14.552048 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:14.551998 2574 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 19 07:42:14.554769 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.554746 2574 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 19 07:42:14.613156 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.613075 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 19 07:42:14.614134 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.614112 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-70.ec2.internal" event="NodeHasSufficientMemory" Jul 19 07:42:14.614242 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.614145 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-70.ec2.internal" event="NodeHasNoDiskPressure" Jul 19 07:42:14.614242 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.614157 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-70.ec2.internal" event="NodeHasSufficientPID" Jul 19 07:42:14.614242 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.614186 2574 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-136-70.ec2.internal" Jul 19 07:42:14.626714 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.626693 2574 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-136-70.ec2.internal" Jul 19 07:42:14.626794 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:14.626719 2574 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-70.ec2.internal\": node \"ip-10-0-136-70.ec2.internal\" not found" Jul 19 07:42:14.645980 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:14.645953 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-70.ec2.internal\" not found" Jul 19 07:42:14.653103 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.653080 2574 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-70.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-70.ec2.internal"] Jul 19 07:42:14.653178 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.653143 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 19 07:42:14.656895 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.656878 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-70.ec2.internal" event="NodeHasSufficientMemory" Jul 19 07:42:14.657003 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.656910 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-70.ec2.internal" event="NodeHasNoDiskPressure" Jul 19 07:42:14.657003 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.656954 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-70.ec2.internal" event="NodeHasSufficientPID" Jul 19 07:42:14.659457 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.659441 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 19 07:42:14.659588 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.659574 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-70.ec2.internal" Jul 19 07:42:14.659650 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.659607 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 19 07:42:14.660238 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.660211 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-70.ec2.internal" event="NodeHasSufficientMemory" Jul 19 07:42:14.660238 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.660236 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-70.ec2.internal" event="NodeHasNoDiskPressure" Jul 19 07:42:14.660341 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.660248 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-70.ec2.internal" event="NodeHasSufficientPID" Jul 19 07:42:14.660341 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.660212 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-70.ec2.internal" event="NodeHasSufficientMemory" Jul 19 07:42:14.660341 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.660281 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-70.ec2.internal" event="NodeHasNoDiskPressure" Jul 19 07:42:14.660341 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.660292 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-70.ec2.internal" event="NodeHasSufficientPID" Jul 19 07:42:14.662589 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.662575 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-70.ec2.internal" Jul 19 07:42:14.662663 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.662602 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 19 07:42:14.663287 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.663270 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-70.ec2.internal" event="NodeHasSufficientMemory" Jul 19 07:42:14.663357 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.663296 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-70.ec2.internal" event="NodeHasNoDiskPressure" Jul 19 07:42:14.663357 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.663313 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-70.ec2.internal" event="NodeHasSufficientPID" Jul 19 07:42:14.685207 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:14.685179 2574 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-70.ec2.internal\" not found" node="ip-10-0-136-70.ec2.internal" Jul 19 07:42:14.688687 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:14.688672 2574 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-70.ec2.internal\" not found" node="ip-10-0-136-70.ec2.internal" Jul 19 07:42:14.746650 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:14.746616 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-70.ec2.internal\" not found" Jul 19 07:42:14.753970 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.753946 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2e9c98df0848b5c4bc3c830d475a22f3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-70.ec2.internal\" (UID: \"2e9c98df0848b5c4bc3c830d475a22f3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-70.ec2.internal" Jul 19 07:42:14.754098 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.753975 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2e9c98df0848b5c4bc3c830d475a22f3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-70.ec2.internal\" (UID: \"2e9c98df0848b5c4bc3c830d475a22f3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-70.ec2.internal" Jul 19 07:42:14.754098 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.753996 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/76b974fafc626c9019b0378af47d7689-config\") pod \"kube-apiserver-proxy-ip-10-0-136-70.ec2.internal\" (UID: \"76b974fafc626c9019b0378af47d7689\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-70.ec2.internal" Jul 19 07:42:14.847128 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:14.847093 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-70.ec2.internal\" not found" Jul 19 07:42:14.854420 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.854399 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2e9c98df0848b5c4bc3c830d475a22f3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-70.ec2.internal\" (UID: \"2e9c98df0848b5c4bc3c830d475a22f3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-70.ec2.internal" Jul 19 07:42:14.854491 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.854433 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2e9c98df0848b5c4bc3c830d475a22f3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-70.ec2.internal\" (UID: \"2e9c98df0848b5c4bc3c830d475a22f3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-70.ec2.internal" Jul 19 07:42:14.854491 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.854456 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/76b974fafc626c9019b0378af47d7689-config\") pod \"kube-apiserver-proxy-ip-10-0-136-70.ec2.internal\" (UID: \"76b974fafc626c9019b0378af47d7689\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-70.ec2.internal" Jul 19 07:42:14.854581 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.854507 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/76b974fafc626c9019b0378af47d7689-config\") pod \"kube-apiserver-proxy-ip-10-0-136-70.ec2.internal\" (UID: \"76b974fafc626c9019b0378af47d7689\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-70.ec2.internal" Jul 19 07:42:14.854581 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.854512 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2e9c98df0848b5c4bc3c830d475a22f3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-70.ec2.internal\" (UID: \"2e9c98df0848b5c4bc3c830d475a22f3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-70.ec2.internal" Jul 19 07:42:14.854581 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.854558 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2e9c98df0848b5c4bc3c830d475a22f3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-70.ec2.internal\" (UID: \"2e9c98df0848b5c4bc3c830d475a22f3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-70.ec2.internal" Jul 19 07:42:14.947960 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:14.947875 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-70.ec2.internal\" not found" Jul 19 07:42:14.987423 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.987398 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-70.ec2.internal" Jul 19 07:42:14.991827 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:14.991809 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-70.ec2.internal" Jul 19 07:42:15.048336 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:15.048306 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-70.ec2.internal\" not found" Jul 19 07:42:15.148806 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:15.148778 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-70.ec2.internal\" not found" Jul 19 07:42:15.249357 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:15.249298 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-70.ec2.internal\" not found" Jul 19 07:42:15.266321 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:15.266294 2574 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 19 07:42:15.349716 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:15.349686 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-70.ec2.internal\" not found" Jul 19 07:42:15.367383 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:15.367361 2574 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 19 07:42:15.384729 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:15.384705 2574 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 19 07:42:15.384853 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:15.384835 2574 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 19 07:42:15.384896 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:15.384871 2574 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 19 07:42:15.384934 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:15.384871 2574 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 19 07:42:15.449807 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:15.449777 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-70.ec2.internal\" not found" Jul 19 07:42:15.477621 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:15.477573 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-18 07:37:14 +0000 UTC" deadline="2028-04-19 17:33:19.784689202 +0000 UTC" Jul 19 07:42:15.477621 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:15.477611 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15369h51m4.30708282s" Jul 19 07:42:15.538604 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:15.538511 2574 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 19 07:42:15.552492 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:15.552466 2574 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-70.ec2.internal" Jul 19 07:42:15.564076 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:15.564054 2574 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 19 07:42:15.565142 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:15.565119 2574 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-70.ec2.internal" Jul 19 07:42:15.573156 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:15.573137 2574 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 19 07:42:15.576702 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:15.576664 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e9c98df0848b5c4bc3c830d475a22f3.slice/crio-cbe67d12fa211617b566cae30fc5825ccf19083799568506991bff73abd95aeb WatchSource:0}: Error finding container cbe67d12fa211617b566cae30fc5825ccf19083799568506991bff73abd95aeb: Status 404 returned error can't find the container with id cbe67d12fa211617b566cae30fc5825ccf19083799568506991bff73abd95aeb Jul 19 07:42:15.577083 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:15.577062 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76b974fafc626c9019b0378af47d7689.slice/crio-9df1c25f7bc41311ddd1ebd4554c3b36a2516cb76f8e728b7d860cfcce88c45f WatchSource:0}: Error finding container 9df1c25f7bc41311ddd1ebd4554c3b36a2516cb76f8e728b7d860cfcce88c45f: Status 404 returned error can't find the container with id 9df1c25f7bc41311ddd1ebd4554c3b36a2516cb76f8e728b7d860cfcce88c45f Jul 19 07:42:15.582772 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:15.582758 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 19 07:42:16.427535 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.427470 2574 apiserver.go:52] "Watching apiserver" Jul 19 07:42:16.436821 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.436794 2574 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 19 07:42:16.438284 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.438252 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-p5wwk","kube-system/konnectivity-agent-bnnwq","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wskg4","openshift-image-registry/node-ca-ctt95","openshift-multus/multus-k76h2","openshift-multus/network-metrics-daemon-4r4zh","openshift-network-diagnostics/network-check-target-lnxd5","openshift-network-operator/iptables-alerter-f5pcm","kube-system/kube-apiserver-proxy-ip-10-0-136-70.ec2.internal","openshift-cluster-node-tuning-operator/tuned-79f6n","openshift-dns/node-resolver-5dh7h","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-70.ec2.internal","openshift-multus/multus-additional-cni-plugins-lpbgg","openshift-ovn-kubernetes/ovnkube-node-nrkg5"] Jul 19 07:42:16.441200 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.441176 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lnxd5" Jul 19 07:42:16.441316 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:16.441260 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lnxd5" podUID="8320ca86-7c1c-45ce-9fe9-ee6d779d5d4d" Jul 19 07:42:16.443226 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.443189 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lpbgg" Jul 19 07:42:16.446657 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.445513 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wskg4" Jul 19 07:42:16.446657 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.446496 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 19 07:42:16.446959 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.446942 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 19 07:42:16.447058 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.446994 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-w5b7k\"" Jul 19 07:42:16.447249 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.447230 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 19 07:42:16.447337 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.447230 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 19 07:42:16.447337 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.447326 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 19 07:42:16.447616 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.447597 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ctt95" Jul 19 07:42:16.447852 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.447700 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 19 07:42:16.448346 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.448079 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 19 07:42:16.448346 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.448083 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-zkqft\"" Jul 19 07:42:16.448560 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.448545 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 19 07:42:16.451137 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.450142 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 19 07:42:16.451137 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.450385 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 19 07:42:16.451137 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.450625 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-k76h2" Jul 19 07:42:16.451339 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.451229 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 19 07:42:16.451506 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.451439 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-7b449\"" Jul 19 07:42:16.453590 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.451808 2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 19 07:42:16.453590 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.452497 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4r4zh" Jul 19 07:42:16.453590 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:16.452584 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4r4zh" podUID="8279f51a-80ab-4d6f-98b0-32805b4d0355" Jul 19 07:42:16.453590 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.452953 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 19 07:42:16.453590 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.453230 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-j6g72\"" Jul 19 07:42:16.457298 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.455172 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-nrkg5" Jul 19 07:42:16.459177 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.457874 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-v6bd7\"" Jul 19 07:42:16.459177 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.457962 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 19 07:42:16.459177 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.458089 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 19 07:42:16.459177 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.458306 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 19 07:42:16.459177 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.458314 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 19 07:42:16.459177 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.458489 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-f5pcm" Jul 19 07:42:16.459177 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.458509 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 19 07:42:16.459177 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.459069 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 19 07:42:16.460498 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.459742 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5dh7h" Jul 19 07:42:16.461443 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.461039 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 19 07:42:16.461443 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.461128 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 19 07:42:16.461443 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.461048 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-nkr9x\"" Jul 19 07:42:16.461676 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.461481 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dba80f1e-6d8a-47c2-a0dc-034f93da275b-multus-cni-dir\") pod \"multus-k76h2\" (UID: \"dba80f1e-6d8a-47c2-a0dc-034f93da275b\") " pod="openshift-multus/multus-k76h2" Jul 19 07:42:16.461676 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.461511 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dba80f1e-6d8a-47c2-a0dc-034f93da275b-hostroot\") pod \"multus-k76h2\" (UID: \"dba80f1e-6d8a-47c2-a0dc-034f93da275b\") " pod="openshift-multus/multus-k76h2" Jul 19 07:42:16.461676 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.461556 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clglj\" (UniqueName: \"kubernetes.io/projected/8320ca86-7c1c-45ce-9fe9-ee6d779d5d4d-kube-api-access-clglj\") pod \"network-check-target-lnxd5\" (UID: \"8320ca86-7c1c-45ce-9fe9-ee6d779d5d4d\") " pod="openshift-network-diagnostics/network-check-target-lnxd5" Jul 19 07:42:16.461676 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.461585 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3e07cf7b-66a0-4756-b8d6-dcaccc94dc52-host-run-ovn-kubernetes\") pod \"ovnkube-node-nrkg5\" (UID: \"3e07cf7b-66a0-4756-b8d6-dcaccc94dc52\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkg5" Jul 19 07:42:16.461676 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.461611 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/dba80f1e-6d8a-47c2-a0dc-034f93da275b-os-release\") pod \"multus-k76h2\" (UID: \"dba80f1e-6d8a-47c2-a0dc-034f93da275b\") " pod="openshift-multus/multus-k76h2" Jul 19 07:42:16.461676 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.461634 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3e07cf7b-66a0-4756-b8d6-dcaccc94dc52-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nrkg5\" (UID: \"3e07cf7b-66a0-4756-b8d6-dcaccc94dc52\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkg5" Jul 19 07:42:16.461676 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.461659 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6d024bd7-3f48-4487-b740-2bb4a4e197fc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lpbgg\" (UID: \"6d024bd7-3f48-4487-b740-2bb4a4e197fc\") " pod="openshift-multus/multus-additional-cni-plugins-lpbgg" Jul 19 07:42:16.462018 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.461686 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3e07cf7b-66a0-4756-b8d6-dcaccc94dc52-var-lib-openvswitch\") pod \"ovnkube-node-nrkg5\" (UID: \"3e07cf7b-66a0-4756-b8d6-dcaccc94dc52\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkg5" Jul 19 07:42:16.462018 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.461711 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3e07cf7b-66a0-4756-b8d6-dcaccc94dc52-ovn-node-metrics-cert\") pod \"ovnkube-node-nrkg5\" (UID: \"3e07cf7b-66a0-4756-b8d6-dcaccc94dc52\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkg5" Jul 19 07:42:16.462018 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.461782 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6d024bd7-3f48-4487-b740-2bb4a4e197fc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lpbgg\" (UID: \"6d024bd7-3f48-4487-b740-2bb4a4e197fc\") " pod="openshift-multus/multus-additional-cni-plugins-lpbgg" Jul 19 07:42:16.462018 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.461820 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c52de7f2-39fd-4f3f-ae50-ef961ed37926-device-dir\") pod \"aws-ebs-csi-driver-node-wskg4\" (UID: \"c52de7f2-39fd-4f3f-ae50-ef961ed37926\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wskg4" Jul 19 07:42:16.462018 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.461844 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dba80f1e-6d8a-47c2-a0dc-034f93da275b-host-run-multus-certs\") pod \"multus-k76h2\" (UID: \"dba80f1e-6d8a-47c2-a0dc-034f93da275b\") " pod="openshift-multus/multus-k76h2" Jul 19 07:42:16.462018 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.461866 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3e07cf7b-66a0-4756-b8d6-dcaccc94dc52-run-systemd\") pod \"ovnkube-node-nrkg5\" (UID: \"3e07cf7b-66a0-4756-b8d6-dcaccc94dc52\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkg5" Jul 19 07:42:16.462018 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.461888 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3e07cf7b-66a0-4756-b8d6-dcaccc94dc52-host-cni-netd\") pod \"ovnkube-node-nrkg5\" (UID: \"3e07cf7b-66a0-4756-b8d6-dcaccc94dc52\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkg5" Jul 19 07:42:16.462018 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.461910 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c52de7f2-39fd-4f3f-ae50-ef961ed37926-socket-dir\") pod \"aws-ebs-csi-driver-node-wskg4\" (UID: \"c52de7f2-39fd-4f3f-ae50-ef961ed37926\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wskg4" Jul 19 07:42:16.462018 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.461931 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dba80f1e-6d8a-47c2-a0dc-034f93da275b-system-cni-dir\") pod \"multus-k76h2\" (UID: \"dba80f1e-6d8a-47c2-a0dc-034f93da275b\") " pod="openshift-multus/multus-k76h2" Jul 19 07:42:16.462018 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.461953 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3e07cf7b-66a0-4756-b8d6-dcaccc94dc52-host-run-netns\") pod \"ovnkube-node-nrkg5\" (UID: \"3e07cf7b-66a0-4756-b8d6-dcaccc94dc52\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkg5" Jul 19 07:42:16.462018 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.461985 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6d024bd7-3f48-4487-b740-2bb4a4e197fc-system-cni-dir\") pod \"multus-additional-cni-plugins-lpbgg\" (UID: \"6d024bd7-3f48-4487-b740-2bb4a4e197fc\") " pod="openshift-multus/multus-additional-cni-plugins-lpbgg" Jul 19 07:42:16.462018 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.462012 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3e07cf7b-66a0-4756-b8d6-dcaccc94dc52-host-slash\") pod \"ovnkube-node-nrkg5\" (UID: \"3e07cf7b-66a0-4756-b8d6-dcaccc94dc52\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkg5" Jul 19 07:42:16.462604 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.462035 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3e07cf7b-66a0-4756-b8d6-dcaccc94dc52-log-socket\") pod \"ovnkube-node-nrkg5\" (UID: \"3e07cf7b-66a0-4756-b8d6-dcaccc94dc52\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkg5" Jul 19 07:42:16.462604 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.462070 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dba80f1e-6d8a-47c2-a0dc-034f93da275b-etc-kubernetes\") pod \"multus-k76h2\" (UID: \"dba80f1e-6d8a-47c2-a0dc-034f93da275b\") " pod="openshift-multus/multus-k76h2" Jul 19 07:42:16.462702 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.462603 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/67825d1c-90c9-4e7f-87c8-575b38afa13a-host\") pod \"node-ca-ctt95\" (UID: \"67825d1c-90c9-4e7f-87c8-575b38afa13a\") " pod="openshift-image-registry/node-ca-ctt95" Jul 19 07:42:16.462702 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.462637 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fmw7\" (UniqueName: \"kubernetes.io/projected/c52de7f2-39fd-4f3f-ae50-ef961ed37926-kube-api-access-4fmw7\") pod \"aws-ebs-csi-driver-node-wskg4\" (UID: \"c52de7f2-39fd-4f3f-ae50-ef961ed37926\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wskg4" Jul 19 07:42:16.462702 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.462670 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dba80f1e-6d8a-47c2-a0dc-034f93da275b-cni-binary-copy\") pod \"multus-k76h2\" (UID: \"dba80f1e-6d8a-47c2-a0dc-034f93da275b\") " pod="openshift-multus/multus-k76h2" Jul 19 07:42:16.462834 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.462708 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dba80f1e-6d8a-47c2-a0dc-034f93da275b-multus-socket-dir-parent\") pod \"multus-k76h2\" (UID: \"dba80f1e-6d8a-47c2-a0dc-034f93da275b\") " pod="openshift-multus/multus-k76h2" Jul 19 07:42:16.462834 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.462759 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sckq2\" (UniqueName: \"kubernetes.io/projected/dba80f1e-6d8a-47c2-a0dc-034f93da275b-kube-api-access-sckq2\") pod \"multus-k76h2\" (UID: \"dba80f1e-6d8a-47c2-a0dc-034f93da275b\") " pod="openshift-multus/multus-k76h2" Jul 19 07:42:16.462834 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.462785 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3e07cf7b-66a0-4756-b8d6-dcaccc94dc52-ovnkube-config\") pod \"ovnkube-node-nrkg5\" (UID: \"3e07cf7b-66a0-4756-b8d6-dcaccc94dc52\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkg5" Jul 19 07:42:16.462834 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.462811 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dba80f1e-6d8a-47c2-a0dc-034f93da275b-host-run-k8s-cni-cncf-io\") pod \"multus-k76h2\" (UID: \"dba80f1e-6d8a-47c2-a0dc-034f93da275b\") " pod="openshift-multus/multus-k76h2" Jul 19 07:42:16.463015 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.462850 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3e07cf7b-66a0-4756-b8d6-dcaccc94dc52-env-overrides\") pod \"ovnkube-node-nrkg5\" (UID: \"3e07cf7b-66a0-4756-b8d6-dcaccc94dc52\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkg5" Jul 19 07:42:16.463015 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.462930 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c52de7f2-39fd-4f3f-ae50-ef961ed37926-registration-dir\") pod \"aws-ebs-csi-driver-node-wskg4\" (UID: \"c52de7f2-39fd-4f3f-ae50-ef961ed37926\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wskg4" Jul 19 07:42:16.463015 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.462986 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9vcz\" (UniqueName: \"kubernetes.io/projected/8279f51a-80ab-4d6f-98b0-32805b4d0355-kube-api-access-d9vcz\") pod \"network-metrics-daemon-4r4zh\" (UID: \"8279f51a-80ab-4d6f-98b0-32805b4d0355\") " pod="openshift-multus/network-metrics-daemon-4r4zh" Jul 19 07:42:16.463173 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.463036 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6d024bd7-3f48-4487-b740-2bb4a4e197fc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lpbgg\" (UID: \"6d024bd7-3f48-4487-b740-2bb4a4e197fc\") " pod="openshift-multus/multus-additional-cni-plugins-lpbgg" Jul 19 07:42:16.463173 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.463097 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c52de7f2-39fd-4f3f-ae50-ef961ed37926-etc-selinux\") pod \"aws-ebs-csi-driver-node-wskg4\" (UID: \"c52de7f2-39fd-4f3f-ae50-ef961ed37926\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wskg4" Jul 19 07:42:16.463173 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.463122 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dba80f1e-6d8a-47c2-a0dc-034f93da275b-host-var-lib-cni-multus\") pod \"multus-k76h2\" (UID: \"dba80f1e-6d8a-47c2-a0dc-034f93da275b\") " pod="openshift-multus/multus-k76h2" Jul 19 07:42:16.463173 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.463146 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kdf8\" (UniqueName: \"kubernetes.io/projected/67825d1c-90c9-4e7f-87c8-575b38afa13a-kube-api-access-2kdf8\") pod \"node-ca-ctt95\" (UID: \"67825d1c-90c9-4e7f-87c8-575b38afa13a\") " pod="openshift-image-registry/node-ca-ctt95" Jul 19 07:42:16.463417 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.463183 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3e07cf7b-66a0-4756-b8d6-dcaccc94dc52-run-openvswitch\") pod \"ovnkube-node-nrkg5\" (UID: \"3e07cf7b-66a0-4756-b8d6-dcaccc94dc52\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkg5" Jul 19 07:42:16.463417 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.463221 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3e07cf7b-66a0-4756-b8d6-dcaccc94dc52-host-cni-bin\") pod \"ovnkube-node-nrkg5\" (UID: \"3e07cf7b-66a0-4756-b8d6-dcaccc94dc52\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkg5" Jul 19 07:42:16.463417 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.463227 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-79f6n" Jul 19 07:42:16.463417 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.463245 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6d024bd7-3f48-4487-b740-2bb4a4e197fc-os-release\") pod \"multus-additional-cni-plugins-lpbgg\" (UID: \"6d024bd7-3f48-4487-b740-2bb4a4e197fc\") " pod="openshift-multus/multus-additional-cni-plugins-lpbgg" Jul 19 07:42:16.463417 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.463267 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 19 07:42:16.464362 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.463270 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dba80f1e-6d8a-47c2-a0dc-034f93da275b-host-var-lib-kubelet\") pod \"multus-k76h2\" (UID: \"dba80f1e-6d8a-47c2-a0dc-034f93da275b\") " pod="openshift-multus/multus-k76h2" Jul 19 07:42:16.464362 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.463458 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3e07cf7b-66a0-4756-b8d6-dcaccc94dc52-systemd-units\") pod \"ovnkube-node-nrkg5\" (UID: \"3e07cf7b-66a0-4756-b8d6-dcaccc94dc52\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkg5" Jul 19 07:42:16.464362 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.463482 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3e07cf7b-66a0-4756-b8d6-dcaccc94dc52-node-log\") pod \"ovnkube-node-nrkg5\" (UID: \"3e07cf7b-66a0-4756-b8d6-dcaccc94dc52\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkg5" Jul 19 07:42:16.464362 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.463504 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7mpz\" (UniqueName: \"kubernetes.io/projected/3e07cf7b-66a0-4756-b8d6-dcaccc94dc52-kube-api-access-j7mpz\") pod \"ovnkube-node-nrkg5\" (UID: \"3e07cf7b-66a0-4756-b8d6-dcaccc94dc52\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkg5" Jul 19 07:42:16.464362 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.463541 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3e07cf7b-66a0-4756-b8d6-dcaccc94dc52-etc-openvswitch\") pod \"ovnkube-node-nrkg5\" (UID: \"3e07cf7b-66a0-4756-b8d6-dcaccc94dc52\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkg5" Jul 19 07:42:16.464362 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.463610 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6d024bd7-3f48-4487-b740-2bb4a4e197fc-cnibin\") pod \"multus-additional-cni-plugins-lpbgg\" (UID: \"6d024bd7-3f48-4487-b740-2bb4a4e197fc\") " pod="openshift-multus/multus-additional-cni-plugins-lpbgg" Jul 19 07:42:16.464362 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.463637 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 19 07:42:16.464362 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.463641 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c52de7f2-39fd-4f3f-ae50-ef961ed37926-sys-fs\") pod \"aws-ebs-csi-driver-node-wskg4\" (UID: \"c52de7f2-39fd-4f3f-ae50-ef961ed37926\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wskg4" Jul 19 07:42:16.464362 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.463685 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dba80f1e-6d8a-47c2-a0dc-034f93da275b-multus-daemon-config\") pod \"multus-k76h2\" (UID: \"dba80f1e-6d8a-47c2-a0dc-034f93da275b\") " pod="openshift-multus/multus-k76h2" Jul 19 07:42:16.464362 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.463708 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/67825d1c-90c9-4e7f-87c8-575b38afa13a-serviceca\") pod \"node-ca-ctt95\" (UID: \"67825d1c-90c9-4e7f-87c8-575b38afa13a\") " pod="openshift-image-registry/node-ca-ctt95" Jul 19 07:42:16.464362 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.463748 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3e07cf7b-66a0-4756-b8d6-dcaccc94dc52-host-kubelet\") pod \"ovnkube-node-nrkg5\" (UID: \"3e07cf7b-66a0-4756-b8d6-dcaccc94dc52\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkg5" Jul 19 07:42:16.464362 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.463782 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3e07cf7b-66a0-4756-b8d6-dcaccc94dc52-ovnkube-script-lib\") pod \"ovnkube-node-nrkg5\" (UID: \"3e07cf7b-66a0-4756-b8d6-dcaccc94dc52\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkg5" Jul 19 07:42:16.464362 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.463810 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dba80f1e-6d8a-47c2-a0dc-034f93da275b-host-run-netns\") pod \"multus-k76h2\" (UID: \"dba80f1e-6d8a-47c2-a0dc-034f93da275b\") " pod="openshift-multus/multus-k76h2" Jul 19 07:42:16.464362 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.463830 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-bnq79\"" Jul 19 07:42:16.464362 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.463841 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6m2sv\" (UniqueName: \"kubernetes.io/projected/6d024bd7-3f48-4487-b740-2bb4a4e197fc-kube-api-access-6m2sv\") pod \"multus-additional-cni-plugins-lpbgg\" (UID: \"6d024bd7-3f48-4487-b740-2bb4a4e197fc\") " pod="openshift-multus/multus-additional-cni-plugins-lpbgg" Jul 19 07:42:16.464362 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.463864 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8279f51a-80ab-4d6f-98b0-32805b4d0355-metrics-certs\") pod \"network-metrics-daemon-4r4zh\" (UID: \"8279f51a-80ab-4d6f-98b0-32805b4d0355\") " pod="openshift-multus/network-metrics-daemon-4r4zh" Jul 19 07:42:16.464362 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.463884 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3e07cf7b-66a0-4756-b8d6-dcaccc94dc52-run-ovn\") pod \"ovnkube-node-nrkg5\" (UID: \"3e07cf7b-66a0-4756-b8d6-dcaccc94dc52\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkg5" Jul 19 07:42:16.465241 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.463907 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6d024bd7-3f48-4487-b740-2bb4a4e197fc-cni-binary-copy\") pod \"multus-additional-cni-plugins-lpbgg\" (UID: \"6d024bd7-3f48-4487-b740-2bb4a4e197fc\") " pod="openshift-multus/multus-additional-cni-plugins-lpbgg" Jul 19 07:42:16.465241 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.463944 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c52de7f2-39fd-4f3f-ae50-ef961ed37926-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wskg4\" (UID: \"c52de7f2-39fd-4f3f-ae50-ef961ed37926\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wskg4" Jul 19 07:42:16.465241 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.464022 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dba80f1e-6d8a-47c2-a0dc-034f93da275b-cnibin\") pod \"multus-k76h2\" (UID: \"dba80f1e-6d8a-47c2-a0dc-034f93da275b\") " pod="openshift-multus/multus-k76h2" Jul 19 07:42:16.465241 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.464047 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dba80f1e-6d8a-47c2-a0dc-034f93da275b-host-var-lib-cni-bin\") pod \"multus-k76h2\" (UID: \"dba80f1e-6d8a-47c2-a0dc-034f93da275b\") " pod="openshift-multus/multus-k76h2" Jul 19 07:42:16.465241 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.464068 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 19 07:42:16.465241 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.464078 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dba80f1e-6d8a-47c2-a0dc-034f93da275b-multus-conf-dir\") pod \"multus-k76h2\" (UID: \"dba80f1e-6d8a-47c2-a0dc-034f93da275b\") " pod="openshift-multus/multus-k76h2" Jul 19 07:42:16.465241 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.465237 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-bnnwq" Jul 19 07:42:16.466112 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.465737 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 19 07:42:16.466112 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.465967 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-5qkv5\"" Jul 19 07:42:16.466362 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.466257 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 19 07:42:16.468352 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.468331 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p5wwk" Jul 19 07:42:16.469433 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:16.468730 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p5wwk" podUID="17c9482e-aaa4-41b0-8aad-ac6f428f1f48" Jul 19 07:42:16.469433 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.468968 2574 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 19 07:42:16.469433 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.469358 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 19 07:42:16.469433 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.469418 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 19 07:42:16.469738 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.469660 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-m46r8\"" Jul 19 07:42:16.483339 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.483318 2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-dwrsk" Jul 19 07:42:16.490695 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.490673 2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-dwrsk" Jul 19 07:42:16.554020 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.553983 2574 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 19 07:42:16.558442 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.558386 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-70.ec2.internal" event={"ID":"76b974fafc626c9019b0378af47d7689","Type":"ContainerStarted","Data":"9df1c25f7bc41311ddd1ebd4554c3b36a2516cb76f8e728b7d860cfcce88c45f"} Jul 19 07:42:16.560799 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.560738 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-70.ec2.internal" event={"ID":"2e9c98df0848b5c4bc3c830d475a22f3","Type":"ContainerStarted","Data":"cbe67d12fa211617b566cae30fc5825ccf19083799568506991bff73abd95aeb"} Jul 19 07:42:16.565133 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.565109 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3e07cf7b-66a0-4756-b8d6-dcaccc94dc52-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nrkg5\" (UID: \"3e07cf7b-66a0-4756-b8d6-dcaccc94dc52\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkg5" Jul 19 07:42:16.565269 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.565151 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/17c9482e-aaa4-41b0-8aad-ac6f428f1f48-kubelet-config\") pod \"global-pull-secret-syncer-p5wwk\" (UID: \"17c9482e-aaa4-41b0-8aad-ac6f428f1f48\") " pod="kube-system/global-pull-secret-syncer-p5wwk" Jul 19 07:42:16.565269 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.565171 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6d024bd7-3f48-4487-b740-2bb4a4e197fc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lpbgg\" (UID: \"6d024bd7-3f48-4487-b740-2bb4a4e197fc\") " pod="openshift-multus/multus-additional-cni-plugins-lpbgg" Jul 19 07:42:16.565269 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.565195 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3e07cf7b-66a0-4756-b8d6-dcaccc94dc52-var-lib-openvswitch\") pod \"ovnkube-node-nrkg5\" (UID: \"3e07cf7b-66a0-4756-b8d6-dcaccc94dc52\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkg5" Jul 19 07:42:16.565269 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.565221 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3e07cf7b-66a0-4756-b8d6-dcaccc94dc52-ovn-node-metrics-cert\") pod \"ovnkube-node-nrkg5\" (UID: \"3e07cf7b-66a0-4756-b8d6-dcaccc94dc52\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkg5" Jul 19 07:42:16.565269 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.565249 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8259d1a2-204c-4d43-a9ce-939a685dc44f-agent-certs\") pod \"konnectivity-agent-bnnwq\" (UID: \"8259d1a2-204c-4d43-a9ce-939a685dc44f\") " pod="kube-system/konnectivity-agent-bnnwq" Jul 19 07:42:16.565269 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.565259 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3e07cf7b-66a0-4756-b8d6-dcaccc94dc52-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nrkg5\" (UID: \"3e07cf7b-66a0-4756-b8d6-dcaccc94dc52\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkg5" Jul 19 07:42:16.565592 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.565274 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/17c9482e-aaa4-41b0-8aad-ac6f428f1f48-original-pull-secret\") pod \"global-pull-secret-syncer-p5wwk\" (UID: \"17c9482e-aaa4-41b0-8aad-ac6f428f1f48\") " pod="kube-system/global-pull-secret-syncer-p5wwk" Jul 19 07:42:16.565592 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.565316 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6d024bd7-3f48-4487-b740-2bb4a4e197fc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lpbgg\" (UID: \"6d024bd7-3f48-4487-b740-2bb4a4e197fc\") " pod="openshift-multus/multus-additional-cni-plugins-lpbgg" Jul 19 07:42:16.565592 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.565336 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3e07cf7b-66a0-4756-b8d6-dcaccc94dc52-var-lib-openvswitch\") pod \"ovnkube-node-nrkg5\" (UID: \"3e07cf7b-66a0-4756-b8d6-dcaccc94dc52\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkg5" Jul 19 07:42:16.565592 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.565349 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c52de7f2-39fd-4f3f-ae50-ef961ed37926-device-dir\") pod \"aws-ebs-csi-driver-node-wskg4\" (UID: \"c52de7f2-39fd-4f3f-ae50-ef961ed37926\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wskg4" Jul 19 07:42:16.565592 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.565375 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dba80f1e-6d8a-47c2-a0dc-034f93da275b-host-run-multus-certs\") pod \"multus-k76h2\" (UID: \"dba80f1e-6d8a-47c2-a0dc-034f93da275b\") " pod="openshift-multus/multus-k76h2" Jul 19 07:42:16.565592 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.565402 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3e07cf7b-66a0-4756-b8d6-dcaccc94dc52-run-systemd\") pod \"ovnkube-node-nrkg5\" (UID: \"3e07cf7b-66a0-4756-b8d6-dcaccc94dc52\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkg5" Jul 19 07:42:16.565592 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.565430 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzbcz\" (UniqueName: \"kubernetes.io/projected/8cea20a0-fc13-42b5-8c65-a67697252f18-kube-api-access-xzbcz\") pod \"iptables-alerter-f5pcm\" (UID: \"8cea20a0-fc13-42b5-8c65-a67697252f18\") " pod="openshift-network-operator/iptables-alerter-f5pcm" Jul 19 07:42:16.565592 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.565455 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3e07cf7b-66a0-4756-b8d6-dcaccc94dc52-host-cni-netd\") pod \"ovnkube-node-nrkg5\" (UID: \"3e07cf7b-66a0-4756-b8d6-dcaccc94dc52\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkg5" Jul 19 07:42:16.565592 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.565480 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c52de7f2-39fd-4f3f-ae50-ef961ed37926-socket-dir\") pod \"aws-ebs-csi-driver-node-wskg4\" (UID: \"c52de7f2-39fd-4f3f-ae50-ef961ed37926\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wskg4" Jul 19 07:42:16.565592 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.565504 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dba80f1e-6d8a-47c2-a0dc-034f93da275b-system-cni-dir\") pod \"multus-k76h2\" (UID: \"dba80f1e-6d8a-47c2-a0dc-034f93da275b\") " pod="openshift-multus/multus-k76h2" Jul 19 07:42:16.565592 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.565564 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3e07cf7b-66a0-4756-b8d6-dcaccc94dc52-host-run-netns\") pod \"ovnkube-node-nrkg5\" (UID: \"3e07cf7b-66a0-4756-b8d6-dcaccc94dc52\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkg5" Jul 19 07:42:16.565592 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.565589 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8cea20a0-fc13-42b5-8c65-a67697252f18-iptables-alerter-script\") pod \"iptables-alerter-f5pcm\" (UID: \"8cea20a0-fc13-42b5-8c65-a67697252f18\") " pod="openshift-network-operator/iptables-alerter-f5pcm" Jul 19 07:42:16.566180 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.565613 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/62c0ff0a-c82e-4d0e-a721-9a49d1d59e11-etc-sysctl-d\") pod \"tuned-79f6n\" (UID: \"62c0ff0a-c82e-4d0e-a721-9a49d1d59e11\") " pod="openshift-cluster-node-tuning-operator/tuned-79f6n" Jul 19 07:42:16.566180 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.565638 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6d024bd7-3f48-4487-b740-2bb4a4e197fc-system-cni-dir\") pod \"multus-additional-cni-plugins-lpbgg\" (UID: \"6d024bd7-3f48-4487-b740-2bb4a4e197fc\") " pod="openshift-multus/multus-additional-cni-plugins-lpbgg" Jul 19 07:42:16.566180 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.565672 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3e07cf7b-66a0-4756-b8d6-dcaccc94dc52-host-slash\") pod \"ovnkube-node-nrkg5\" (UID: \"3e07cf7b-66a0-4756-b8d6-dcaccc94dc52\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkg5" Jul 19 07:42:16.566180 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.565698 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3e07cf7b-66a0-4756-b8d6-dcaccc94dc52-log-socket\") pod \"ovnkube-node-nrkg5\" (UID: \"3e07cf7b-66a0-4756-b8d6-dcaccc94dc52\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkg5" Jul 19 07:42:16.566180 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.565734 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cecbc684-8b2f-4c00-a7ac-2966f0d95873-hosts-file\") pod \"node-resolver-5dh7h\" (UID: \"cecbc684-8b2f-4c00-a7ac-2966f0d95873\") " pod="openshift-dns/node-resolver-5dh7h" Jul 19 07:42:16.566180 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.565736 2574 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 19 07:42:16.566180 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.565759 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dba80f1e-6d8a-47c2-a0dc-034f93da275b-etc-kubernetes\") pod \"multus-k76h2\" (UID: \"dba80f1e-6d8a-47c2-a0dc-034f93da275b\") " pod="openshift-multus/multus-k76h2" Jul 19 07:42:16.566180 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.565786 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/67825d1c-90c9-4e7f-87c8-575b38afa13a-host\") pod \"node-ca-ctt95\" (UID: \"67825d1c-90c9-4e7f-87c8-575b38afa13a\") " pod="openshift-image-registry/node-ca-ctt95" Jul 19 07:42:16.566180 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.565811 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fmw7\" (UniqueName: \"kubernetes.io/projected/c52de7f2-39fd-4f3f-ae50-ef961ed37926-kube-api-access-4fmw7\") pod \"aws-ebs-csi-driver-node-wskg4\" (UID: \"c52de7f2-39fd-4f3f-ae50-ef961ed37926\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wskg4" Jul 19 07:42:16.566180 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.565839 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dba80f1e-6d8a-47c2-a0dc-034f93da275b-cni-binary-copy\") pod \"multus-k76h2\" (UID: \"dba80f1e-6d8a-47c2-a0dc-034f93da275b\") " pod="openshift-multus/multus-k76h2" Jul 19 07:42:16.566180 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.565869 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dba80f1e-6d8a-47c2-a0dc-034f93da275b-multus-socket-dir-parent\") pod \"multus-k76h2\" (UID: \"dba80f1e-6d8a-47c2-a0dc-034f93da275b\") " pod="openshift-multus/multus-k76h2" Jul 19 07:42:16.566180 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.565880 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6d024bd7-3f48-4487-b740-2bb4a4e197fc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lpbgg\" (UID: \"6d024bd7-3f48-4487-b740-2bb4a4e197fc\") " pod="openshift-multus/multus-additional-cni-plugins-lpbgg" Jul 19 07:42:16.566180 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.565915 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sckq2\" (UniqueName: \"kubernetes.io/projected/dba80f1e-6d8a-47c2-a0dc-034f93da275b-kube-api-access-sckq2\") pod \"multus-k76h2\" (UID: \"dba80f1e-6d8a-47c2-a0dc-034f93da275b\") " pod="openshift-multus/multus-k76h2" Jul 19 07:42:16.566180 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.565953 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3e07cf7b-66a0-4756-b8d6-dcaccc94dc52-ovnkube-config\") pod \"ovnkube-node-nrkg5\" (UID: \"3e07cf7b-66a0-4756-b8d6-dcaccc94dc52\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkg5" Jul 19 07:42:16.566180 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.565977 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dba80f1e-6d8a-47c2-a0dc-034f93da275b-host-run-k8s-cni-cncf-io\") pod \"multus-k76h2\" (UID: \"dba80f1e-6d8a-47c2-a0dc-034f93da275b\") " pod="openshift-multus/multus-k76h2" Jul 19 07:42:16.566180 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.566001 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3e07cf7b-66a0-4756-b8d6-dcaccc94dc52-env-overrides\") pod \"ovnkube-node-nrkg5\" (UID: \"3e07cf7b-66a0-4756-b8d6-dcaccc94dc52\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkg5" Jul 19 07:42:16.566180 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.566029 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/62c0ff0a-c82e-4d0e-a721-9a49d1d59e11-etc-sysconfig\") pod \"tuned-79f6n\" (UID: \"62c0ff0a-c82e-4d0e-a721-9a49d1d59e11\") " pod="openshift-cluster-node-tuning-operator/tuned-79f6n" Jul 19 07:42:16.566992 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.566053 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/62c0ff0a-c82e-4d0e-a721-9a49d1d59e11-run\") pod \"tuned-79f6n\" (UID: \"62c0ff0a-c82e-4d0e-a721-9a49d1d59e11\") " pod="openshift-cluster-node-tuning-operator/tuned-79f6n" Jul 19 07:42:16.566992 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.566077 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/62c0ff0a-c82e-4d0e-a721-9a49d1d59e11-etc-tuned\") pod \"tuned-79f6n\" (UID: \"62c0ff0a-c82e-4d0e-a721-9a49d1d59e11\") " pod="openshift-cluster-node-tuning-operator/tuned-79f6n" Jul 19 07:42:16.566992 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.566095 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c52de7f2-39fd-4f3f-ae50-ef961ed37926-device-dir\") pod \"aws-ebs-csi-driver-node-wskg4\" (UID: \"c52de7f2-39fd-4f3f-ae50-ef961ed37926\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wskg4" Jul 19 07:42:16.566992 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.566099 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhxgw\" (UniqueName: \"kubernetes.io/projected/62c0ff0a-c82e-4d0e-a721-9a49d1d59e11-kube-api-access-dhxgw\") pod \"tuned-79f6n\" (UID: \"62c0ff0a-c82e-4d0e-a721-9a49d1d59e11\") " pod="openshift-cluster-node-tuning-operator/tuned-79f6n" Jul 19 07:42:16.566992 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.566159 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dba80f1e-6d8a-47c2-a0dc-034f93da275b-host-run-multus-certs\") pod \"multus-k76h2\" (UID: \"dba80f1e-6d8a-47c2-a0dc-034f93da275b\") " pod="openshift-multus/multus-k76h2" Jul 19 07:42:16.566992 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.566221 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dba80f1e-6d8a-47c2-a0dc-034f93da275b-etc-kubernetes\") pod \"multus-k76h2\" (UID: \"dba80f1e-6d8a-47c2-a0dc-034f93da275b\") " pod="openshift-multus/multus-k76h2" Jul 19 07:42:16.566992 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.566278 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dba80f1e-6d8a-47c2-a0dc-034f93da275b-host-run-k8s-cni-cncf-io\") pod \"multus-k76h2\" (UID: \"dba80f1e-6d8a-47c2-a0dc-034f93da275b\") " pod="openshift-multus/multus-k76h2" Jul 19 07:42:16.566992 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.566423 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/67825d1c-90c9-4e7f-87c8-575b38afa13a-host\") pod \"node-ca-ctt95\" (UID: \"67825d1c-90c9-4e7f-87c8-575b38afa13a\") " pod="openshift-image-registry/node-ca-ctt95" Jul 19 07:42:16.566992 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.566499 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3e07cf7b-66a0-4756-b8d6-dcaccc94dc52-host-cni-netd\") pod \"ovnkube-node-nrkg5\" (UID: \"3e07cf7b-66a0-4756-b8d6-dcaccc94dc52\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkg5" Jul 19 07:42:16.566992 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.566627 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c52de7f2-39fd-4f3f-ae50-ef961ed37926-socket-dir\") pod \"aws-ebs-csi-driver-node-wskg4\" (UID: \"c52de7f2-39fd-4f3f-ae50-ef961ed37926\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wskg4" Jul 19 07:42:16.566992 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.566685 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dba80f1e-6d8a-47c2-a0dc-034f93da275b-system-cni-dir\") pod \"multus-k76h2\" (UID: \"dba80f1e-6d8a-47c2-a0dc-034f93da275b\") " pod="openshift-multus/multus-k76h2" Jul 19 07:42:16.566992 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.566714 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3e07cf7b-66a0-4756-b8d6-dcaccc94dc52-env-overrides\") pod \"ovnkube-node-nrkg5\" (UID: \"3e07cf7b-66a0-4756-b8d6-dcaccc94dc52\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkg5" Jul 19 07:42:16.566992 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.566721 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6d024bd7-3f48-4487-b740-2bb4a4e197fc-system-cni-dir\") pod \"multus-additional-cni-plugins-lpbgg\" (UID: \"6d024bd7-3f48-4487-b740-2bb4a4e197fc\") " pod="openshift-multus/multus-additional-cni-plugins-lpbgg" Jul 19 07:42:16.566992 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.566750 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3e07cf7b-66a0-4756-b8d6-dcaccc94dc52-host-slash\") pod \"ovnkube-node-nrkg5\" (UID: \"3e07cf7b-66a0-4756-b8d6-dcaccc94dc52\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkg5" Jul 19 07:42:16.566992 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.566050 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6d024bd7-3f48-4487-b740-2bb4a4e197fc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lpbgg\" (UID: \"6d024bd7-3f48-4487-b740-2bb4a4e197fc\") " pod="openshift-multus/multus-additional-cni-plugins-lpbgg" Jul 19 07:42:16.566992 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.566759 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dba80f1e-6d8a-47c2-a0dc-034f93da275b-cni-binary-copy\") pod \"multus-k76h2\" (UID: \"dba80f1e-6d8a-47c2-a0dc-034f93da275b\") " pod="openshift-multus/multus-k76h2" Jul 19 07:42:16.566992 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.566798 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3e07cf7b-66a0-4756-b8d6-dcaccc94dc52-log-socket\") pod \"ovnkube-node-nrkg5\" (UID: \"3e07cf7b-66a0-4756-b8d6-dcaccc94dc52\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkg5" Jul 19 07:42:16.566992 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.566802 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3e07cf7b-66a0-4756-b8d6-dcaccc94dc52-host-run-netns\") pod \"ovnkube-node-nrkg5\" (UID: \"3e07cf7b-66a0-4756-b8d6-dcaccc94dc52\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkg5" Jul 19 07:42:16.567905 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.566798 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3e07cf7b-66a0-4756-b8d6-dcaccc94dc52-run-systemd\") pod \"ovnkube-node-nrkg5\" (UID: \"3e07cf7b-66a0-4756-b8d6-dcaccc94dc52\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkg5" Jul 19 07:42:16.567905 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.566820 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dba80f1e-6d8a-47c2-a0dc-034f93da275b-multus-socket-dir-parent\") pod \"multus-k76h2\" (UID: \"dba80f1e-6d8a-47c2-a0dc-034f93da275b\") " pod="openshift-multus/multus-k76h2" Jul 19 07:42:16.567905 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.566823 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/17c9482e-aaa4-41b0-8aad-ac6f428f1f48-dbus\") pod \"global-pull-secret-syncer-p5wwk\" (UID: \"17c9482e-aaa4-41b0-8aad-ac6f428f1f48\") " pod="kube-system/global-pull-secret-syncer-p5wwk" Jul 19 07:42:16.567905 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.566900 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c52de7f2-39fd-4f3f-ae50-ef961ed37926-registration-dir\") pod \"aws-ebs-csi-driver-node-wskg4\" (UID: \"c52de7f2-39fd-4f3f-ae50-ef961ed37926\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wskg4" Jul 19 07:42:16.567905 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.566933 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9vcz\" (UniqueName: \"kubernetes.io/projected/8279f51a-80ab-4d6f-98b0-32805b4d0355-kube-api-access-d9vcz\") pod \"network-metrics-daemon-4r4zh\" (UID: \"8279f51a-80ab-4d6f-98b0-32805b4d0355\") " pod="openshift-multus/network-metrics-daemon-4r4zh" Jul 19 07:42:16.567905 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.566950 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3e07cf7b-66a0-4756-b8d6-dcaccc94dc52-ovnkube-config\") pod \"ovnkube-node-nrkg5\" (UID: \"3e07cf7b-66a0-4756-b8d6-dcaccc94dc52\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkg5" Jul 19 07:42:16.567905 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.566963 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cecbc684-8b2f-4c00-a7ac-2966f0d95873-tmp-dir\") pod \"node-resolver-5dh7h\" (UID: \"cecbc684-8b2f-4c00-a7ac-2966f0d95873\") " pod="openshift-dns/node-resolver-5dh7h" Jul 19 07:42:16.567905 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.567005 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6d024bd7-3f48-4487-b740-2bb4a4e197fc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lpbgg\" (UID: \"6d024bd7-3f48-4487-b740-2bb4a4e197fc\") " pod="openshift-multus/multus-additional-cni-plugins-lpbgg" Jul 19 07:42:16.567905 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.567015 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c52de7f2-39fd-4f3f-ae50-ef961ed37926-registration-dir\") pod \"aws-ebs-csi-driver-node-wskg4\" (UID: \"c52de7f2-39fd-4f3f-ae50-ef961ed37926\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wskg4" Jul 19 07:42:16.567905 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.567035 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c52de7f2-39fd-4f3f-ae50-ef961ed37926-etc-selinux\") pod \"aws-ebs-csi-driver-node-wskg4\" (UID: \"c52de7f2-39fd-4f3f-ae50-ef961ed37926\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wskg4" Jul 19 07:42:16.567905 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.567087 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dba80f1e-6d8a-47c2-a0dc-034f93da275b-host-var-lib-cni-multus\") pod \"multus-k76h2\" (UID: \"dba80f1e-6d8a-47c2-a0dc-034f93da275b\") " pod="openshift-multus/multus-k76h2" Jul 19 07:42:16.567905 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.567099 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c52de7f2-39fd-4f3f-ae50-ef961ed37926-etc-selinux\") pod \"aws-ebs-csi-driver-node-wskg4\" (UID: \"c52de7f2-39fd-4f3f-ae50-ef961ed37926\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wskg4" Jul 19 07:42:16.567905 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.567121 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kdf8\" (UniqueName: \"kubernetes.io/projected/67825d1c-90c9-4e7f-87c8-575b38afa13a-kube-api-access-2kdf8\") pod \"node-ca-ctt95\" (UID: \"67825d1c-90c9-4e7f-87c8-575b38afa13a\") " pod="openshift-image-registry/node-ca-ctt95" Jul 19 07:42:16.567905 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.567145 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3e07cf7b-66a0-4756-b8d6-dcaccc94dc52-run-openvswitch\") pod \"ovnkube-node-nrkg5\" (UID: \"3e07cf7b-66a0-4756-b8d6-dcaccc94dc52\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkg5" Jul 19 07:42:16.567905 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.567186 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3e07cf7b-66a0-4756-b8d6-dcaccc94dc52-host-cni-bin\") pod \"ovnkube-node-nrkg5\" (UID: \"3e07cf7b-66a0-4756-b8d6-dcaccc94dc52\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkg5" Jul 19 07:42:16.567905 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.567196 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dba80f1e-6d8a-47c2-a0dc-034f93da275b-host-var-lib-cni-multus\") pod \"multus-k76h2\" (UID: \"dba80f1e-6d8a-47c2-a0dc-034f93da275b\") " pod="openshift-multus/multus-k76h2" Jul 19 07:42:16.567905 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.567217 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/62c0ff0a-c82e-4d0e-a721-9a49d1d59e11-etc-sysctl-conf\") pod \"tuned-79f6n\" (UID: \"62c0ff0a-c82e-4d0e-a721-9a49d1d59e11\") " pod="openshift-cluster-node-tuning-operator/tuned-79f6n" Jul 19 07:42:16.568734 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.567235 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3e07cf7b-66a0-4756-b8d6-dcaccc94dc52-run-openvswitch\") pod \"ovnkube-node-nrkg5\" (UID: \"3e07cf7b-66a0-4756-b8d6-dcaccc94dc52\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkg5" Jul 19 07:42:16.568734 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.567242 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/62c0ff0a-c82e-4d0e-a721-9a49d1d59e11-etc-systemd\") pod \"tuned-79f6n\" (UID: \"62c0ff0a-c82e-4d0e-a721-9a49d1d59e11\") " pod="openshift-cluster-node-tuning-operator/tuned-79f6n" Jul 19 07:42:16.568734 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.567267 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6d024bd7-3f48-4487-b740-2bb4a4e197fc-os-release\") pod \"multus-additional-cni-plugins-lpbgg\" (UID: \"6d024bd7-3f48-4487-b740-2bb4a4e197fc\") " pod="openshift-multus/multus-additional-cni-plugins-lpbgg" Jul 19 07:42:16.568734 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.567276 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3e07cf7b-66a0-4756-b8d6-dcaccc94dc52-host-cni-bin\") pod \"ovnkube-node-nrkg5\" (UID: \"3e07cf7b-66a0-4756-b8d6-dcaccc94dc52\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkg5" Jul 19 07:42:16.568734 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.567407 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6d024bd7-3f48-4487-b740-2bb4a4e197fc-os-release\") pod \"multus-additional-cni-plugins-lpbgg\" (UID: \"6d024bd7-3f48-4487-b740-2bb4a4e197fc\") " pod="openshift-multus/multus-additional-cni-plugins-lpbgg" Jul 19 07:42:16.568734 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.567735 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dba80f1e-6d8a-47c2-a0dc-034f93da275b-host-var-lib-kubelet\") pod \"multus-k76h2\" (UID: \"dba80f1e-6d8a-47c2-a0dc-034f93da275b\") " pod="openshift-multus/multus-k76h2" Jul 19 07:42:16.568734 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.567764 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3e07cf7b-66a0-4756-b8d6-dcaccc94dc52-systemd-units\") pod \"ovnkube-node-nrkg5\" (UID: \"3e07cf7b-66a0-4756-b8d6-dcaccc94dc52\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkg5" Jul 19 07:42:16.568734 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.567789 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3e07cf7b-66a0-4756-b8d6-dcaccc94dc52-node-log\") pod \"ovnkube-node-nrkg5\" (UID: \"3e07cf7b-66a0-4756-b8d6-dcaccc94dc52\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkg5" Jul 19 07:42:16.568734 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.567813 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7mpz\" (UniqueName: \"kubernetes.io/projected/3e07cf7b-66a0-4756-b8d6-dcaccc94dc52-kube-api-access-j7mpz\") pod \"ovnkube-node-nrkg5\" (UID: \"3e07cf7b-66a0-4756-b8d6-dcaccc94dc52\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkg5" Jul 19 07:42:16.568734 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.567843 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/62c0ff0a-c82e-4d0e-a721-9a49d1d59e11-sys\") pod \"tuned-79f6n\" (UID: \"62c0ff0a-c82e-4d0e-a721-9a49d1d59e11\") " pod="openshift-cluster-node-tuning-operator/tuned-79f6n" Jul 19 07:42:16.568734 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.567872 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/62c0ff0a-c82e-4d0e-a721-9a49d1d59e11-tmp\") pod \"tuned-79f6n\" (UID: \"62c0ff0a-c82e-4d0e-a721-9a49d1d59e11\") " pod="openshift-cluster-node-tuning-operator/tuned-79f6n" Jul 19 07:42:16.568734 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.567895 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8259d1a2-204c-4d43-a9ce-939a685dc44f-konnectivity-ca\") pod \"konnectivity-agent-bnnwq\" (UID: \"8259d1a2-204c-4d43-a9ce-939a685dc44f\") " pod="kube-system/konnectivity-agent-bnnwq" Jul 19 07:42:16.568734 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.567931 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3e07cf7b-66a0-4756-b8d6-dcaccc94dc52-etc-openvswitch\") pod \"ovnkube-node-nrkg5\" (UID: \"3e07cf7b-66a0-4756-b8d6-dcaccc94dc52\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkg5" Jul 19 07:42:16.568734 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.567936 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dba80f1e-6d8a-47c2-a0dc-034f93da275b-host-var-lib-kubelet\") pod \"multus-k76h2\" (UID: \"dba80f1e-6d8a-47c2-a0dc-034f93da275b\") " pod="openshift-multus/multus-k76h2" Jul 19 07:42:16.568734 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.567976 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6d024bd7-3f48-4487-b740-2bb4a4e197fc-cnibin\") pod \"multus-additional-cni-plugins-lpbgg\" (UID: \"6d024bd7-3f48-4487-b740-2bb4a4e197fc\") " pod="openshift-multus/multus-additional-cni-plugins-lpbgg" Jul 19 07:42:16.568734 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.567981 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3e07cf7b-66a0-4756-b8d6-dcaccc94dc52-etc-openvswitch\") pod \"ovnkube-node-nrkg5\" (UID: \"3e07cf7b-66a0-4756-b8d6-dcaccc94dc52\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkg5" Jul 19 07:42:16.568734 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.568004 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c52de7f2-39fd-4f3f-ae50-ef961ed37926-sys-fs\") pod \"aws-ebs-csi-driver-node-wskg4\" (UID: \"c52de7f2-39fd-4f3f-ae50-ef961ed37926\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wskg4" Jul 19 07:42:16.569534 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.568032 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dba80f1e-6d8a-47c2-a0dc-034f93da275b-multus-daemon-config\") pod \"multus-k76h2\" (UID: \"dba80f1e-6d8a-47c2-a0dc-034f93da275b\") " pod="openshift-multus/multus-k76h2" Jul 19 07:42:16.569534 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.568059 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/67825d1c-90c9-4e7f-87c8-575b38afa13a-serviceca\") pod \"node-ca-ctt95\" (UID: \"67825d1c-90c9-4e7f-87c8-575b38afa13a\") " pod="openshift-image-registry/node-ca-ctt95" Jul 19 07:42:16.569534 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.568082 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3e07cf7b-66a0-4756-b8d6-dcaccc94dc52-host-kubelet\") pod \"ovnkube-node-nrkg5\" (UID: \"3e07cf7b-66a0-4756-b8d6-dcaccc94dc52\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkg5" Jul 19 07:42:16.569534 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.568104 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3e07cf7b-66a0-4756-b8d6-dcaccc94dc52-ovnkube-script-lib\") pod \"ovnkube-node-nrkg5\" (UID: \"3e07cf7b-66a0-4756-b8d6-dcaccc94dc52\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkg5" Jul 19 07:42:16.569534 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.568126 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6d024bd7-3f48-4487-b740-2bb4a4e197fc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lpbgg\" (UID: \"6d024bd7-3f48-4487-b740-2bb4a4e197fc\") " pod="openshift-multus/multus-additional-cni-plugins-lpbgg" Jul 19 07:42:16.569534 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.568185 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6d024bd7-3f48-4487-b740-2bb4a4e197fc-cnibin\") pod \"multus-additional-cni-plugins-lpbgg\" (UID: \"6d024bd7-3f48-4487-b740-2bb4a4e197fc\") " pod="openshift-multus/multus-additional-cni-plugins-lpbgg" Jul 19 07:42:16.569534 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.568187 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c52de7f2-39fd-4f3f-ae50-ef961ed37926-sys-fs\") pod \"aws-ebs-csi-driver-node-wskg4\" (UID: \"c52de7f2-39fd-4f3f-ae50-ef961ed37926\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wskg4" Jul 19 07:42:16.569534 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.568130 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/62c0ff0a-c82e-4d0e-a721-9a49d1d59e11-etc-modprobe-d\") pod \"tuned-79f6n\" (UID: \"62c0ff0a-c82e-4d0e-a721-9a49d1d59e11\") " pod="openshift-cluster-node-tuning-operator/tuned-79f6n" Jul 19 07:42:16.569534 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.568234 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dba80f1e-6d8a-47c2-a0dc-034f93da275b-host-run-netns\") pod \"multus-k76h2\" (UID: \"dba80f1e-6d8a-47c2-a0dc-034f93da275b\") " pod="openshift-multus/multus-k76h2" Jul 19 07:42:16.569534 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.568027 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3e07cf7b-66a0-4756-b8d6-dcaccc94dc52-node-log\") pod \"ovnkube-node-nrkg5\" (UID: \"3e07cf7b-66a0-4756-b8d6-dcaccc94dc52\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkg5" Jul 19 07:42:16.569534 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.567893 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3e07cf7b-66a0-4756-b8d6-dcaccc94dc52-systemd-units\") pod \"ovnkube-node-nrkg5\" (UID: \"3e07cf7b-66a0-4756-b8d6-dcaccc94dc52\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkg5" Jul 19 07:42:16.569534 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.568276 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8cea20a0-fc13-42b5-8c65-a67697252f18-host-slash\") pod \"iptables-alerter-f5pcm\" (UID: \"8cea20a0-fc13-42b5-8c65-a67697252f18\") " pod="openshift-network-operator/iptables-alerter-f5pcm" Jul 19 07:42:16.569534 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.568324 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/62c0ff0a-c82e-4d0e-a721-9a49d1d59e11-lib-modules\") pod \"tuned-79f6n\" (UID: \"62c0ff0a-c82e-4d0e-a721-9a49d1d59e11\") " pod="openshift-cluster-node-tuning-operator/tuned-79f6n" Jul 19 07:42:16.569534 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.568355 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m2sv\" (UniqueName: \"kubernetes.io/projected/6d024bd7-3f48-4487-b740-2bb4a4e197fc-kube-api-access-6m2sv\") pod \"multus-additional-cni-plugins-lpbgg\" (UID: \"6d024bd7-3f48-4487-b740-2bb4a4e197fc\") " pod="openshift-multus/multus-additional-cni-plugins-lpbgg" Jul 19 07:42:16.569534 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.568325 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dba80f1e-6d8a-47c2-a0dc-034f93da275b-host-run-netns\") pod \"multus-k76h2\" (UID: \"dba80f1e-6d8a-47c2-a0dc-034f93da275b\") " pod="openshift-multus/multus-k76h2" Jul 19 07:42:16.569534 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.568413 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8279f51a-80ab-4d6f-98b0-32805b4d0355-metrics-certs\") pod \"network-metrics-daemon-4r4zh\" (UID: \"8279f51a-80ab-4d6f-98b0-32805b4d0355\") " pod="openshift-multus/network-metrics-daemon-4r4zh" Jul 19 07:42:16.569534 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.568437 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3e07cf7b-66a0-4756-b8d6-dcaccc94dc52-run-ovn\") pod \"ovnkube-node-nrkg5\" (UID: \"3e07cf7b-66a0-4756-b8d6-dcaccc94dc52\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkg5" Jul 19 07:42:16.570347 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.568460 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/62c0ff0a-c82e-4d0e-a721-9a49d1d59e11-host\") pod \"tuned-79f6n\" (UID: \"62c0ff0a-c82e-4d0e-a721-9a49d1d59e11\") " pod="openshift-cluster-node-tuning-operator/tuned-79f6n" Jul 19 07:42:16.570347 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.568486 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6d024bd7-3f48-4487-b740-2bb4a4e197fc-cni-binary-copy\") pod \"multus-additional-cni-plugins-lpbgg\" (UID: \"6d024bd7-3f48-4487-b740-2bb4a4e197fc\") " pod="openshift-multus/multus-additional-cni-plugins-lpbgg" Jul 19 07:42:16.570347 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.568545 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c52de7f2-39fd-4f3f-ae50-ef961ed37926-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wskg4\" (UID: \"c52de7f2-39fd-4f3f-ae50-ef961ed37926\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wskg4" Jul 19 07:42:16.570347 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.568572 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dba80f1e-6d8a-47c2-a0dc-034f93da275b-cnibin\") pod \"multus-k76h2\" (UID: \"dba80f1e-6d8a-47c2-a0dc-034f93da275b\") " pod="openshift-multus/multus-k76h2" Jul 19 07:42:16.570347 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.568648 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dba80f1e-6d8a-47c2-a0dc-034f93da275b-host-var-lib-cni-bin\") pod \"multus-k76h2\" (UID: \"dba80f1e-6d8a-47c2-a0dc-034f93da275b\") " pod="openshift-multus/multus-k76h2" Jul 19 07:42:16.570347 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.568661 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3e07cf7b-66a0-4756-b8d6-dcaccc94dc52-host-kubelet\") pod \"ovnkube-node-nrkg5\" (UID: \"3e07cf7b-66a0-4756-b8d6-dcaccc94dc52\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkg5" Jul 19 07:42:16.570347 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.568674 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dba80f1e-6d8a-47c2-a0dc-034f93da275b-multus-conf-dir\") pod \"multus-k76h2\" (UID: \"dba80f1e-6d8a-47c2-a0dc-034f93da275b\") " pod="openshift-multus/multus-k76h2" Jul 19 07:42:16.570347 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.568667 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dba80f1e-6d8a-47c2-a0dc-034f93da275b-multus-daemon-config\") pod \"multus-k76h2\" (UID: \"dba80f1e-6d8a-47c2-a0dc-034f93da275b\") " pod="openshift-multus/multus-k76h2" Jul 19 07:42:16.570347 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.568698 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/62c0ff0a-c82e-4d0e-a721-9a49d1d59e11-etc-kubernetes\") pod \"tuned-79f6n\" (UID: \"62c0ff0a-c82e-4d0e-a721-9a49d1d59e11\") " pod="openshift-cluster-node-tuning-operator/tuned-79f6n" Jul 19 07:42:16.570347 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.568720 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/62c0ff0a-c82e-4d0e-a721-9a49d1d59e11-var-lib-kubelet\") pod \"tuned-79f6n\" (UID: \"62c0ff0a-c82e-4d0e-a721-9a49d1d59e11\") " pod="openshift-cluster-node-tuning-operator/tuned-79f6n" Jul 19 07:42:16.570347 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.568725 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/67825d1c-90c9-4e7f-87c8-575b38afa13a-serviceca\") pod \"node-ca-ctt95\" (UID: \"67825d1c-90c9-4e7f-87c8-575b38afa13a\") " pod="openshift-image-registry/node-ca-ctt95" Jul 19 07:42:16.570347 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:16.568735 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 19 07:42:16.570347 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.568744 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dba80f1e-6d8a-47c2-a0dc-034f93da275b-multus-cni-dir\") pod \"multus-k76h2\" (UID: \"dba80f1e-6d8a-47c2-a0dc-034f93da275b\") " pod="openshift-multus/multus-k76h2" Jul 19 07:42:16.570347 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.568766 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dba80f1e-6d8a-47c2-a0dc-034f93da275b-hostroot\") pod \"multus-k76h2\" (UID: \"dba80f1e-6d8a-47c2-a0dc-034f93da275b\") " pod="openshift-multus/multus-k76h2" Jul 19 07:42:16.570347 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.568769 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3e07cf7b-66a0-4756-b8d6-dcaccc94dc52-run-ovn\") pod \"ovnkube-node-nrkg5\" (UID: \"3e07cf7b-66a0-4756-b8d6-dcaccc94dc52\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkg5" Jul 19 07:42:16.570347 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:16.568796 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8279f51a-80ab-4d6f-98b0-32805b4d0355-metrics-certs podName:8279f51a-80ab-4d6f-98b0-32805b4d0355 nodeName:}" failed. No retries permitted until 2026-07-19 07:42:17.06877714 +0000 UTC m=+3.423086741 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8279f51a-80ab-4d6f-98b0-32805b4d0355-metrics-certs") pod "network-metrics-daemon-4r4zh" (UID: "8279f51a-80ab-4d6f-98b0-32805b4d0355") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 19 07:42:16.570347 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.568818 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clglj\" (UniqueName: \"kubernetes.io/projected/8320ca86-7c1c-45ce-9fe9-ee6d779d5d4d-kube-api-access-clglj\") pod \"network-check-target-lnxd5\" (UID: \"8320ca86-7c1c-45ce-9fe9-ee6d779d5d4d\") " pod="openshift-network-diagnostics/network-check-target-lnxd5" Jul 19 07:42:16.571155 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.568844 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3e07cf7b-66a0-4756-b8d6-dcaccc94dc52-host-run-ovn-kubernetes\") pod \"ovnkube-node-nrkg5\" (UID: \"3e07cf7b-66a0-4756-b8d6-dcaccc94dc52\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkg5" Jul 19 07:42:16.571155 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.568864 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c52de7f2-39fd-4f3f-ae50-ef961ed37926-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wskg4\" (UID: \"c52de7f2-39fd-4f3f-ae50-ef961ed37926\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wskg4" Jul 19 07:42:16.571155 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.568869 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zb5dc\" (UniqueName: \"kubernetes.io/projected/cecbc684-8b2f-4c00-a7ac-2966f0d95873-kube-api-access-zb5dc\") pod \"node-resolver-5dh7h\" (UID: \"cecbc684-8b2f-4c00-a7ac-2966f0d95873\") " pod="openshift-dns/node-resolver-5dh7h" Jul 19 07:42:16.571155 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.568899 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/dba80f1e-6d8a-47c2-a0dc-034f93da275b-os-release\") pod \"multus-k76h2\" (UID: \"dba80f1e-6d8a-47c2-a0dc-034f93da275b\") " pod="openshift-multus/multus-k76h2" Jul 19 07:42:16.571155 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.568992 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dba80f1e-6d8a-47c2-a0dc-034f93da275b-cnibin\") pod \"multus-k76h2\" (UID: \"dba80f1e-6d8a-47c2-a0dc-034f93da275b\") " pod="openshift-multus/multus-k76h2" Jul 19 07:42:16.571155 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.569008 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/dba80f1e-6d8a-47c2-a0dc-034f93da275b-os-release\") pod \"multus-k76h2\" (UID: \"dba80f1e-6d8a-47c2-a0dc-034f93da275b\") " pod="openshift-multus/multus-k76h2" Jul 19 07:42:16.571155 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.569096 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dba80f1e-6d8a-47c2-a0dc-034f93da275b-hostroot\") pod \"multus-k76h2\" (UID: \"dba80f1e-6d8a-47c2-a0dc-034f93da275b\") " pod="openshift-multus/multus-k76h2" Jul 19 07:42:16.571155 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.569133 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dba80f1e-6d8a-47c2-a0dc-034f93da275b-multus-cni-dir\") pod \"multus-k76h2\" (UID: \"dba80f1e-6d8a-47c2-a0dc-034f93da275b\") " pod="openshift-multus/multus-k76h2" Jul 19 07:42:16.571155 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.568822 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dba80f1e-6d8a-47c2-a0dc-034f93da275b-multus-conf-dir\") pod \"multus-k76h2\" (UID: \"dba80f1e-6d8a-47c2-a0dc-034f93da275b\") " pod="openshift-multus/multus-k76h2" Jul 19 07:42:16.571155 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.569148 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3e07cf7b-66a0-4756-b8d6-dcaccc94dc52-host-run-ovn-kubernetes\") pod \"ovnkube-node-nrkg5\" (UID: \"3e07cf7b-66a0-4756-b8d6-dcaccc94dc52\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkg5" Jul 19 07:42:16.571155 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.569156 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6d024bd7-3f48-4487-b740-2bb4a4e197fc-cni-binary-copy\") pod \"multus-additional-cni-plugins-lpbgg\" (UID: \"6d024bd7-3f48-4487-b740-2bb4a4e197fc\") " pod="openshift-multus/multus-additional-cni-plugins-lpbgg" Jul 19 07:42:16.571155 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.569188 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3e07cf7b-66a0-4756-b8d6-dcaccc94dc52-ovnkube-script-lib\") pod \"ovnkube-node-nrkg5\" (UID: \"3e07cf7b-66a0-4756-b8d6-dcaccc94dc52\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkg5" Jul 19 07:42:16.571155 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.569209 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dba80f1e-6d8a-47c2-a0dc-034f93da275b-host-var-lib-cni-bin\") pod \"multus-k76h2\" (UID: \"dba80f1e-6d8a-47c2-a0dc-034f93da275b\") " pod="openshift-multus/multus-k76h2" Jul 19 07:42:16.574858 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:16.574835 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 19 07:42:16.574963 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:16.574862 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 19 07:42:16.574963 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:16.574875 2574 projected.go:196] Error preparing data for projected volume kube-api-access-clglj for pod openshift-network-diagnostics/network-check-target-lnxd5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 19 07:42:16.574963 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:16.574935 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8320ca86-7c1c-45ce-9fe9-ee6d779d5d4d-kube-api-access-clglj podName:8320ca86-7c1c-45ce-9fe9-ee6d779d5d4d nodeName:}" failed. No retries permitted until 2026-07-19 07:42:17.074916321 +0000 UTC m=+3.429225918 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-clglj" (UniqueName: "kubernetes.io/projected/8320ca86-7c1c-45ce-9fe9-ee6d779d5d4d-kube-api-access-clglj") pod "network-check-target-lnxd5" (UID: "8320ca86-7c1c-45ce-9fe9-ee6d779d5d4d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 19 07:42:16.576295 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.576175 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3e07cf7b-66a0-4756-b8d6-dcaccc94dc52-ovn-node-metrics-cert\") pod \"ovnkube-node-nrkg5\" (UID: \"3e07cf7b-66a0-4756-b8d6-dcaccc94dc52\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkg5" Jul 19 07:42:16.576295 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.576226 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sckq2\" (UniqueName: \"kubernetes.io/projected/dba80f1e-6d8a-47c2-a0dc-034f93da275b-kube-api-access-sckq2\") pod \"multus-k76h2\" (UID: \"dba80f1e-6d8a-47c2-a0dc-034f93da275b\") " pod="openshift-multus/multus-k76h2" Jul 19 07:42:16.576740 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.576693 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kdf8\" (UniqueName: \"kubernetes.io/projected/67825d1c-90c9-4e7f-87c8-575b38afa13a-kube-api-access-2kdf8\") pod \"node-ca-ctt95\" (UID: \"67825d1c-90c9-4e7f-87c8-575b38afa13a\") " pod="openshift-image-registry/node-ca-ctt95" Jul 19 07:42:16.578952 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.578925 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fmw7\" (UniqueName: \"kubernetes.io/projected/c52de7f2-39fd-4f3f-ae50-ef961ed37926-kube-api-access-4fmw7\") pod \"aws-ebs-csi-driver-node-wskg4\" (UID: \"c52de7f2-39fd-4f3f-ae50-ef961ed37926\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wskg4" Jul 19 07:42:16.580177 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.579189 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7mpz\" (UniqueName: \"kubernetes.io/projected/3e07cf7b-66a0-4756-b8d6-dcaccc94dc52-kube-api-access-j7mpz\") pod \"ovnkube-node-nrkg5\" (UID: \"3e07cf7b-66a0-4756-b8d6-dcaccc94dc52\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkg5" Jul 19 07:42:16.580177 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.579261 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9vcz\" (UniqueName: \"kubernetes.io/projected/8279f51a-80ab-4d6f-98b0-32805b4d0355-kube-api-access-d9vcz\") pod \"network-metrics-daemon-4r4zh\" (UID: \"8279f51a-80ab-4d6f-98b0-32805b4d0355\") " pod="openshift-multus/network-metrics-daemon-4r4zh" Jul 19 07:42:16.580177 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.580134 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m2sv\" (UniqueName: \"kubernetes.io/projected/6d024bd7-3f48-4487-b740-2bb4a4e197fc-kube-api-access-6m2sv\") pod \"multus-additional-cni-plugins-lpbgg\" (UID: \"6d024bd7-3f48-4487-b740-2bb4a4e197fc\") " pod="openshift-multus/multus-additional-cni-plugins-lpbgg" Jul 19 07:42:16.669798 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.669759 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/62c0ff0a-c82e-4d0e-a721-9a49d1d59e11-host\") pod \"tuned-79f6n\" (UID: \"62c0ff0a-c82e-4d0e-a721-9a49d1d59e11\") " pod="openshift-cluster-node-tuning-operator/tuned-79f6n" Jul 19 07:42:16.669993 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.669808 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/62c0ff0a-c82e-4d0e-a721-9a49d1d59e11-etc-kubernetes\") pod \"tuned-79f6n\" (UID: \"62c0ff0a-c82e-4d0e-a721-9a49d1d59e11\") " pod="openshift-cluster-node-tuning-operator/tuned-79f6n" Jul 19 07:42:16.669993 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.669829 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/62c0ff0a-c82e-4d0e-a721-9a49d1d59e11-var-lib-kubelet\") pod \"tuned-79f6n\" (UID: \"62c0ff0a-c82e-4d0e-a721-9a49d1d59e11\") " pod="openshift-cluster-node-tuning-operator/tuned-79f6n" Jul 19 07:42:16.669993 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.669865 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb5dc\" (UniqueName: \"kubernetes.io/projected/cecbc684-8b2f-4c00-a7ac-2966f0d95873-kube-api-access-zb5dc\") pod \"node-resolver-5dh7h\" (UID: \"cecbc684-8b2f-4c00-a7ac-2966f0d95873\") " pod="openshift-dns/node-resolver-5dh7h" Jul 19 07:42:16.669993 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.669886 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/17c9482e-aaa4-41b0-8aad-ac6f428f1f48-kubelet-config\") pod \"global-pull-secret-syncer-p5wwk\" (UID: \"17c9482e-aaa4-41b0-8aad-ac6f428f1f48\") " pod="kube-system/global-pull-secret-syncer-p5wwk" Jul 19 07:42:16.669993 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.669892 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/62c0ff0a-c82e-4d0e-a721-9a49d1d59e11-host\") pod \"tuned-79f6n\" (UID: \"62c0ff0a-c82e-4d0e-a721-9a49d1d59e11\") " pod="openshift-cluster-node-tuning-operator/tuned-79f6n" Jul 19 07:42:16.669993 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.669912 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8259d1a2-204c-4d43-a9ce-939a685dc44f-agent-certs\") pod \"konnectivity-agent-bnnwq\" (UID: \"8259d1a2-204c-4d43-a9ce-939a685dc44f\") " pod="kube-system/konnectivity-agent-bnnwq" Jul 19 07:42:16.669993 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.669912 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/62c0ff0a-c82e-4d0e-a721-9a49d1d59e11-etc-kubernetes\") pod \"tuned-79f6n\" (UID: \"62c0ff0a-c82e-4d0e-a721-9a49d1d59e11\") " pod="openshift-cluster-node-tuning-operator/tuned-79f6n" Jul 19 07:42:16.669993 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.669916 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/62c0ff0a-c82e-4d0e-a721-9a49d1d59e11-var-lib-kubelet\") pod \"tuned-79f6n\" (UID: \"62c0ff0a-c82e-4d0e-a721-9a49d1d59e11\") " pod="openshift-cluster-node-tuning-operator/tuned-79f6n" Jul 19 07:42:16.669993 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.669932 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/17c9482e-aaa4-41b0-8aad-ac6f428f1f48-original-pull-secret\") pod \"global-pull-secret-syncer-p5wwk\" (UID: \"17c9482e-aaa4-41b0-8aad-ac6f428f1f48\") " pod="kube-system/global-pull-secret-syncer-p5wwk" Jul 19 07:42:16.669993 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.669965 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/17c9482e-aaa4-41b0-8aad-ac6f428f1f48-kubelet-config\") pod \"global-pull-secret-syncer-p5wwk\" (UID: \"17c9482e-aaa4-41b0-8aad-ac6f428f1f48\") " pod="kube-system/global-pull-secret-syncer-p5wwk" Jul 19 07:42:16.669993 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.669981 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzbcz\" (UniqueName: \"kubernetes.io/projected/8cea20a0-fc13-42b5-8c65-a67697252f18-kube-api-access-xzbcz\") pod \"iptables-alerter-f5pcm\" (UID: \"8cea20a0-fc13-42b5-8c65-a67697252f18\") " pod="openshift-network-operator/iptables-alerter-f5pcm" Jul 19 07:42:16.670453 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.670012 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8cea20a0-fc13-42b5-8c65-a67697252f18-iptables-alerter-script\") pod \"iptables-alerter-f5pcm\" (UID: \"8cea20a0-fc13-42b5-8c65-a67697252f18\") " pod="openshift-network-operator/iptables-alerter-f5pcm" Jul 19 07:42:16.670453 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.670038 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/62c0ff0a-c82e-4d0e-a721-9a49d1d59e11-etc-sysctl-d\") pod \"tuned-79f6n\" (UID: \"62c0ff0a-c82e-4d0e-a721-9a49d1d59e11\") " pod="openshift-cluster-node-tuning-operator/tuned-79f6n" Jul 19 07:42:16.670453 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.670153 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cecbc684-8b2f-4c00-a7ac-2966f0d95873-hosts-file\") pod \"node-resolver-5dh7h\" (UID: \"cecbc684-8b2f-4c00-a7ac-2966f0d95873\") " pod="openshift-dns/node-resolver-5dh7h" Jul 19 07:42:16.670453 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.670189 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/62c0ff0a-c82e-4d0e-a721-9a49d1d59e11-etc-sysctl-d\") pod \"tuned-79f6n\" (UID: \"62c0ff0a-c82e-4d0e-a721-9a49d1d59e11\") " pod="openshift-cluster-node-tuning-operator/tuned-79f6n" Jul 19 07:42:16.670453 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.670201 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/62c0ff0a-c82e-4d0e-a721-9a49d1d59e11-etc-sysconfig\") pod \"tuned-79f6n\" (UID: \"62c0ff0a-c82e-4d0e-a721-9a49d1d59e11\") " pod="openshift-cluster-node-tuning-operator/tuned-79f6n" Jul 19 07:42:16.670453 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.670219 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cecbc684-8b2f-4c00-a7ac-2966f0d95873-hosts-file\") pod \"node-resolver-5dh7h\" (UID: \"cecbc684-8b2f-4c00-a7ac-2966f0d95873\") " pod="openshift-dns/node-resolver-5dh7h" Jul 19 07:42:16.670453 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.670226 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/62c0ff0a-c82e-4d0e-a721-9a49d1d59e11-run\") pod \"tuned-79f6n\" (UID: \"62c0ff0a-c82e-4d0e-a721-9a49d1d59e11\") " pod="openshift-cluster-node-tuning-operator/tuned-79f6n" Jul 19 07:42:16.670453 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:16.670016 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 19 07:42:16.670453 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.670252 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/62c0ff0a-c82e-4d0e-a721-9a49d1d59e11-etc-tuned\") pod \"tuned-79f6n\" (UID: \"62c0ff0a-c82e-4d0e-a721-9a49d1d59e11\") " pod="openshift-cluster-node-tuning-operator/tuned-79f6n" Jul 19 07:42:16.670453 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.670276 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/62c0ff0a-c82e-4d0e-a721-9a49d1d59e11-run\") pod \"tuned-79f6n\" (UID: \"62c0ff0a-c82e-4d0e-a721-9a49d1d59e11\") " pod="openshift-cluster-node-tuning-operator/tuned-79f6n" Jul 19 07:42:16.670453 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.670278 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhxgw\" (UniqueName: \"kubernetes.io/projected/62c0ff0a-c82e-4d0e-a721-9a49d1d59e11-kube-api-access-dhxgw\") pod \"tuned-79f6n\" (UID: \"62c0ff0a-c82e-4d0e-a721-9a49d1d59e11\") " pod="openshift-cluster-node-tuning-operator/tuned-79f6n" Jul 19 07:42:16.670453 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.670242 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/62c0ff0a-c82e-4d0e-a721-9a49d1d59e11-etc-sysconfig\") pod \"tuned-79f6n\" (UID: \"62c0ff0a-c82e-4d0e-a721-9a49d1d59e11\") " pod="openshift-cluster-node-tuning-operator/tuned-79f6n" Jul 19 07:42:16.670453 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.670307 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/17c9482e-aaa4-41b0-8aad-ac6f428f1f48-dbus\") pod \"global-pull-secret-syncer-p5wwk\" (UID: \"17c9482e-aaa4-41b0-8aad-ac6f428f1f48\") " pod="kube-system/global-pull-secret-syncer-p5wwk" Jul 19 07:42:16.670453 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:16.670331 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17c9482e-aaa4-41b0-8aad-ac6f428f1f48-original-pull-secret podName:17c9482e-aaa4-41b0-8aad-ac6f428f1f48 nodeName:}" failed. No retries permitted until 2026-07-19 07:42:17.170304254 +0000 UTC m=+3.524613855 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/17c9482e-aaa4-41b0-8aad-ac6f428f1f48-original-pull-secret") pod "global-pull-secret-syncer-p5wwk" (UID: "17c9482e-aaa4-41b0-8aad-ac6f428f1f48") : object "kube-system"/"original-pull-secret" not registered Jul 19 07:42:16.670453 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.670411 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cecbc684-8b2f-4c00-a7ac-2966f0d95873-tmp-dir\") pod \"node-resolver-5dh7h\" (UID: \"cecbc684-8b2f-4c00-a7ac-2966f0d95873\") " pod="openshift-dns/node-resolver-5dh7h" Jul 19 07:42:16.670453 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.670445 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/62c0ff0a-c82e-4d0e-a721-9a49d1d59e11-etc-sysctl-conf\") pod \"tuned-79f6n\" (UID: \"62c0ff0a-c82e-4d0e-a721-9a49d1d59e11\") " pod="openshift-cluster-node-tuning-operator/tuned-79f6n" Jul 19 07:42:16.671219 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.670467 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/62c0ff0a-c82e-4d0e-a721-9a49d1d59e11-etc-systemd\") pod \"tuned-79f6n\" (UID: \"62c0ff0a-c82e-4d0e-a721-9a49d1d59e11\") " pod="openshift-cluster-node-tuning-operator/tuned-79f6n" Jul 19 07:42:16.671219 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.670495 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/62c0ff0a-c82e-4d0e-a721-9a49d1d59e11-sys\") pod \"tuned-79f6n\" (UID: \"62c0ff0a-c82e-4d0e-a721-9a49d1d59e11\") " pod="openshift-cluster-node-tuning-operator/tuned-79f6n" Jul 19 07:42:16.671219 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.670513 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/17c9482e-aaa4-41b0-8aad-ac6f428f1f48-dbus\") pod \"global-pull-secret-syncer-p5wwk\" (UID: \"17c9482e-aaa4-41b0-8aad-ac6f428f1f48\") " pod="kube-system/global-pull-secret-syncer-p5wwk" Jul 19 07:42:16.671219 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.670533 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/62c0ff0a-c82e-4d0e-a721-9a49d1d59e11-tmp\") pod \"tuned-79f6n\" (UID: \"62c0ff0a-c82e-4d0e-a721-9a49d1d59e11\") " pod="openshift-cluster-node-tuning-operator/tuned-79f6n" Jul 19 07:42:16.671219 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.670587 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8259d1a2-204c-4d43-a9ce-939a685dc44f-konnectivity-ca\") pod \"konnectivity-agent-bnnwq\" (UID: \"8259d1a2-204c-4d43-a9ce-939a685dc44f\") " pod="kube-system/konnectivity-agent-bnnwq" Jul 19 07:42:16.671219 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.670626 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/62c0ff0a-c82e-4d0e-a721-9a49d1d59e11-etc-modprobe-d\") pod \"tuned-79f6n\" (UID: \"62c0ff0a-c82e-4d0e-a721-9a49d1d59e11\") " pod="openshift-cluster-node-tuning-operator/tuned-79f6n" Jul 19 07:42:16.671219 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.670634 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8cea20a0-fc13-42b5-8c65-a67697252f18-iptables-alerter-script\") pod \"iptables-alerter-f5pcm\" (UID: \"8cea20a0-fc13-42b5-8c65-a67697252f18\") " pod="openshift-network-operator/iptables-alerter-f5pcm" Jul 19 07:42:16.671219 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.670654 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8cea20a0-fc13-42b5-8c65-a67697252f18-host-slash\") pod \"iptables-alerter-f5pcm\" (UID: \"8cea20a0-fc13-42b5-8c65-a67697252f18\") " pod="openshift-network-operator/iptables-alerter-f5pcm" Jul 19 07:42:16.671219 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.670680 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/62c0ff0a-c82e-4d0e-a721-9a49d1d59e11-lib-modules\") pod \"tuned-79f6n\" (UID: \"62c0ff0a-c82e-4d0e-a721-9a49d1d59e11\") " pod="openshift-cluster-node-tuning-operator/tuned-79f6n" Jul 19 07:42:16.671219 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.670706 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/62c0ff0a-c82e-4d0e-a721-9a49d1d59e11-sys\") pod \"tuned-79f6n\" (UID: \"62c0ff0a-c82e-4d0e-a721-9a49d1d59e11\") " pod="openshift-cluster-node-tuning-operator/tuned-79f6n" Jul 19 07:42:16.671219 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.670703 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/62c0ff0a-c82e-4d0e-a721-9a49d1d59e11-etc-sysctl-conf\") pod \"tuned-79f6n\" (UID: \"62c0ff0a-c82e-4d0e-a721-9a49d1d59e11\") " pod="openshift-cluster-node-tuning-operator/tuned-79f6n" Jul 19 07:42:16.671219 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.670748 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8cea20a0-fc13-42b5-8c65-a67697252f18-host-slash\") pod \"iptables-alerter-f5pcm\" (UID: \"8cea20a0-fc13-42b5-8c65-a67697252f18\") " pod="openshift-network-operator/iptables-alerter-f5pcm" Jul 19 07:42:16.671219 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.670782 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cecbc684-8b2f-4c00-a7ac-2966f0d95873-tmp-dir\") pod \"node-resolver-5dh7h\" (UID: \"cecbc684-8b2f-4c00-a7ac-2966f0d95873\") " pod="openshift-dns/node-resolver-5dh7h" Jul 19 07:42:16.671219 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.670832 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/62c0ff0a-c82e-4d0e-a721-9a49d1d59e11-etc-modprobe-d\") pod \"tuned-79f6n\" (UID: \"62c0ff0a-c82e-4d0e-a721-9a49d1d59e11\") " pod="openshift-cluster-node-tuning-operator/tuned-79f6n" Jul 19 07:42:16.671219 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.670832 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/62c0ff0a-c82e-4d0e-a721-9a49d1d59e11-etc-systemd\") pod \"tuned-79f6n\" (UID: \"62c0ff0a-c82e-4d0e-a721-9a49d1d59e11\") " pod="openshift-cluster-node-tuning-operator/tuned-79f6n" Jul 19 07:42:16.671219 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.670864 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/62c0ff0a-c82e-4d0e-a721-9a49d1d59e11-lib-modules\") pod \"tuned-79f6n\" (UID: \"62c0ff0a-c82e-4d0e-a721-9a49d1d59e11\") " pod="openshift-cluster-node-tuning-operator/tuned-79f6n" Jul 19 07:42:16.671982 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.671307 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8259d1a2-204c-4d43-a9ce-939a685dc44f-konnectivity-ca\") pod \"konnectivity-agent-bnnwq\" (UID: \"8259d1a2-204c-4d43-a9ce-939a685dc44f\") " pod="kube-system/konnectivity-agent-bnnwq" Jul 19 07:42:16.672869 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.672820 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/62c0ff0a-c82e-4d0e-a721-9a49d1d59e11-etc-tuned\") pod \"tuned-79f6n\" (UID: \"62c0ff0a-c82e-4d0e-a721-9a49d1d59e11\") " pod="openshift-cluster-node-tuning-operator/tuned-79f6n" Jul 19 07:42:16.672869 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.672829 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8259d1a2-204c-4d43-a9ce-939a685dc44f-agent-certs\") pod \"konnectivity-agent-bnnwq\" (UID: \"8259d1a2-204c-4d43-a9ce-939a685dc44f\") " pod="kube-system/konnectivity-agent-bnnwq" Jul 19 07:42:16.673056 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.672881 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/62c0ff0a-c82e-4d0e-a721-9a49d1d59e11-tmp\") pod \"tuned-79f6n\" (UID: \"62c0ff0a-c82e-4d0e-a721-9a49d1d59e11\") " pod="openshift-cluster-node-tuning-operator/tuned-79f6n" Jul 19 07:42:16.680320 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.680252 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzbcz\" (UniqueName: \"kubernetes.io/projected/8cea20a0-fc13-42b5-8c65-a67697252f18-kube-api-access-xzbcz\") pod \"iptables-alerter-f5pcm\" (UID: \"8cea20a0-fc13-42b5-8c65-a67697252f18\") " pod="openshift-network-operator/iptables-alerter-f5pcm" Jul 19 07:42:16.680430 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.680404 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhxgw\" (UniqueName: \"kubernetes.io/projected/62c0ff0a-c82e-4d0e-a721-9a49d1d59e11-kube-api-access-dhxgw\") pod \"tuned-79f6n\" (UID: \"62c0ff0a-c82e-4d0e-a721-9a49d1d59e11\") " pod="openshift-cluster-node-tuning-operator/tuned-79f6n" Jul 19 07:42:16.680675 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.680649 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb5dc\" (UniqueName: \"kubernetes.io/projected/cecbc684-8b2f-4c00-a7ac-2966f0d95873-kube-api-access-zb5dc\") pod \"node-resolver-5dh7h\" (UID: \"cecbc684-8b2f-4c00-a7ac-2966f0d95873\") " pod="openshift-dns/node-resolver-5dh7h" Jul 19 07:42:16.760799 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.760756 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lpbgg" Jul 19 07:42:16.772433 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.772403 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wskg4" Jul 19 07:42:16.782241 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.782218 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ctt95" Jul 19 07:42:16.787966 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.787932 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-k76h2" Jul 19 07:42:16.795739 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.795714 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-nrkg5" Jul 19 07:42:16.803550 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.803510 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-f5pcm" Jul 19 07:42:16.810212 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.810192 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5dh7h" Jul 19 07:42:16.818806 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.818786 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-79f6n" Jul 19 07:42:16.824442 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.824422 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-bnnwq" Jul 19 07:42:16.936294 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:16.936227 2574 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 19 07:42:17.073280 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:17.073239 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8279f51a-80ab-4d6f-98b0-32805b4d0355-metrics-certs\") pod \"network-metrics-daemon-4r4zh\" (UID: \"8279f51a-80ab-4d6f-98b0-32805b4d0355\") " pod="openshift-multus/network-metrics-daemon-4r4zh" Jul 19 07:42:17.073474 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:17.073370 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 19 07:42:17.073474 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:17.073450 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8279f51a-80ab-4d6f-98b0-32805b4d0355-metrics-certs podName:8279f51a-80ab-4d6f-98b0-32805b4d0355 nodeName:}" failed. No retries permitted until 2026-07-19 07:42:18.073429645 +0000 UTC m=+4.427739230 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8279f51a-80ab-4d6f-98b0-32805b4d0355-metrics-certs") pod "network-metrics-daemon-4r4zh" (UID: "8279f51a-80ab-4d6f-98b0-32805b4d0355") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 19 07:42:17.173900 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:17.173861 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clglj\" (UniqueName: \"kubernetes.io/projected/8320ca86-7c1c-45ce-9fe9-ee6d779d5d4d-kube-api-access-clglj\") pod \"network-check-target-lnxd5\" (UID: \"8320ca86-7c1c-45ce-9fe9-ee6d779d5d4d\") " pod="openshift-network-diagnostics/network-check-target-lnxd5" Jul 19 07:42:17.173900 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:17.173905 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/17c9482e-aaa4-41b0-8aad-ac6f428f1f48-original-pull-secret\") pod \"global-pull-secret-syncer-p5wwk\" (UID: \"17c9482e-aaa4-41b0-8aad-ac6f428f1f48\") " pod="kube-system/global-pull-secret-syncer-p5wwk" Jul 19 07:42:17.174127 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:17.174043 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 19 07:42:17.174127 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:17.174048 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 19 07:42:17.174127 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:17.174074 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 19 07:42:17.174127 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:17.174087 2574 projected.go:196] Error preparing data for projected volume kube-api-access-clglj for pod openshift-network-diagnostics/network-check-target-lnxd5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 19 07:42:17.174127 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:17.174110 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17c9482e-aaa4-41b0-8aad-ac6f428f1f48-original-pull-secret podName:17c9482e-aaa4-41b0-8aad-ac6f428f1f48 nodeName:}" failed. No retries permitted until 2026-07-19 07:42:18.17409253 +0000 UTC m=+4.528402109 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/17c9482e-aaa4-41b0-8aad-ac6f428f1f48-original-pull-secret") pod "global-pull-secret-syncer-p5wwk" (UID: "17c9482e-aaa4-41b0-8aad-ac6f428f1f48") : object "kube-system"/"original-pull-secret" not registered Jul 19 07:42:17.174307 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:17.174140 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8320ca86-7c1c-45ce-9fe9-ee6d779d5d4d-kube-api-access-clglj podName:8320ca86-7c1c-45ce-9fe9-ee6d779d5d4d nodeName:}" failed. No retries permitted until 2026-07-19 07:42:18.174121967 +0000 UTC m=+4.528431562 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-clglj" (UniqueName: "kubernetes.io/projected/8320ca86-7c1c-45ce-9fe9-ee6d779d5d4d-kube-api-access-clglj") pod "network-check-target-lnxd5" (UID: "8320ca86-7c1c-45ce-9fe9-ee6d779d5d4d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 19 07:42:17.336823 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:17.336736 2574 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 19 07:42:17.493933 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:17.493891 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-18 07:37:16 +0000 UTC" deadline="2028-01-21 19:58:30.099623493 +0000 UTC" Jul 19 07:42:17.493933 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:17.493928 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13236h16m12.60569915s" Jul 19 07:42:17.552397 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:17.552366 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lnxd5" Jul 19 07:42:17.552595 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:17.552367 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p5wwk" Jul 19 07:42:17.552595 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:17.552498 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lnxd5" podUID="8320ca86-7c1c-45ce-9fe9-ee6d779d5d4d" Jul 19 07:42:17.552595 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:17.552576 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p5wwk" podUID="17c9482e-aaa4-41b0-8aad-ac6f428f1f48" Jul 19 07:42:17.566401 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:17.566375 2574 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 19 07:42:17.606941 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:17.606908 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc52de7f2_39fd_4f3f_ae50_ef961ed37926.slice/crio-e9541d57ab0cf254fe5fce5d94d673ad8ae3236f1fdd3825ca7d8c34461a9315 WatchSource:0}: Error finding container e9541d57ab0cf254fe5fce5d94d673ad8ae3236f1fdd3825ca7d8c34461a9315: Status 404 returned error can't find the container with id e9541d57ab0cf254fe5fce5d94d673ad8ae3236f1fdd3825ca7d8c34461a9315 Jul 19 07:42:17.612443 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:17.612402 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62c0ff0a_c82e_4d0e_a721_9a49d1d59e11.slice/crio-c7ce1fe57b7b7592c1a342b8cc45a5482fe37d08048746a58985f1bbda4d7e9c WatchSource:0}: Error finding container c7ce1fe57b7b7592c1a342b8cc45a5482fe37d08048746a58985f1bbda4d7e9c: Status 404 returned error can't find the container with id c7ce1fe57b7b7592c1a342b8cc45a5482fe37d08048746a58985f1bbda4d7e9c Jul 19 07:42:17.614265 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:17.614242 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddba80f1e_6d8a_47c2_a0dc_034f93da275b.slice/crio-f3e9957abae13628ebec18a699e3abfea8fa027eab596c17ca5db725b4129078 WatchSource:0}: Error finding container f3e9957abae13628ebec18a699e3abfea8fa027eab596c17ca5db725b4129078: Status 404 returned error can't find the container with id f3e9957abae13628ebec18a699e3abfea8fa027eab596c17ca5db725b4129078 Jul 19 07:42:17.617073 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:17.617049 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e07cf7b_66a0_4756_b8d6_dcaccc94dc52.slice/crio-ade52d958857296be0edd22b1e4a8631a59ec8b809e0a7005f3cb2ecbcfb1c93 WatchSource:0}: Error finding container ade52d958857296be0edd22b1e4a8631a59ec8b809e0a7005f3cb2ecbcfb1c93: Status 404 returned error can't find the container with id ade52d958857296be0edd22b1e4a8631a59ec8b809e0a7005f3cb2ecbcfb1c93 Jul 19 07:42:17.618137 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:17.618108 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d024bd7_3f48_4487_b740_2bb4a4e197fc.slice/crio-ee363777e3de6b487fb1b10a0d8e59d8f8ad765c9c0ffc496e1e66839e941db3 WatchSource:0}: Error finding container ee363777e3de6b487fb1b10a0d8e59d8f8ad765c9c0ffc496e1e66839e941db3: Status 404 returned error can't find the container with id ee363777e3de6b487fb1b10a0d8e59d8f8ad765c9c0ffc496e1e66839e941db3 Jul 19 07:42:17.619067 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:17.618874 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67825d1c_90c9_4e7f_87c8_575b38afa13a.slice/crio-19ad32232894efe94059acd25fb8ad1ed692ecd73f883d7f6fd6c25cb681cec8 WatchSource:0}: Error finding container 19ad32232894efe94059acd25fb8ad1ed692ecd73f883d7f6fd6c25cb681cec8: Status 404 returned error can't find the container with id 19ad32232894efe94059acd25fb8ad1ed692ecd73f883d7f6fd6c25cb681cec8 Jul 19 07:42:17.619507 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:17.619481 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cea20a0_fc13_42b5_8c65_a67697252f18.slice/crio-323c142f85f5692ed7c8269d0e351918a5fe244d9a237ec5c0bf4d5f1cb09c0a WatchSource:0}: Error finding container 323c142f85f5692ed7c8269d0e351918a5fe244d9a237ec5c0bf4d5f1cb09c0a: Status 404 returned error can't find the container with id 323c142f85f5692ed7c8269d0e351918a5fe244d9a237ec5c0bf4d5f1cb09c0a Jul 19 07:42:17.642891 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:17.642868 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8259d1a2_204c_4d43_a9ce_939a685dc44f.slice/crio-007558a4c66acbe4a4600fff4b42ba9ed8cec0ad09dd48f35c85081e317ca403 WatchSource:0}: Error finding container 007558a4c66acbe4a4600fff4b42ba9ed8cec0ad09dd48f35c85081e317ca403: Status 404 returned error can't find the container with id 007558a4c66acbe4a4600fff4b42ba9ed8cec0ad09dd48f35c85081e317ca403 Jul 19 07:42:17.643735 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:17.643715 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcecbc684_8b2f_4c00_a7ac_2966f0d95873.slice/crio-fb7516c01f33f99d57a5d69a2b097df21f61c029c32ea3eb160bcf3922831008 WatchSource:0}: Error finding container fb7516c01f33f99d57a5d69a2b097df21f61c029c32ea3eb160bcf3922831008: Status 404 returned error can't find the container with id fb7516c01f33f99d57a5d69a2b097df21f61c029c32ea3eb160bcf3922831008 Jul 19 07:42:18.080462 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:18.080429 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8279f51a-80ab-4d6f-98b0-32805b4d0355-metrics-certs\") pod \"network-metrics-daemon-4r4zh\" (UID: \"8279f51a-80ab-4d6f-98b0-32805b4d0355\") " pod="openshift-multus/network-metrics-daemon-4r4zh" Jul 19 07:42:18.080667 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:18.080579 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 19 07:42:18.080667 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:18.080638 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8279f51a-80ab-4d6f-98b0-32805b4d0355-metrics-certs podName:8279f51a-80ab-4d6f-98b0-32805b4d0355 nodeName:}" failed. No retries permitted until 2026-07-19 07:42:20.080624465 +0000 UTC m=+6.434934045 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8279f51a-80ab-4d6f-98b0-32805b4d0355-metrics-certs") pod "network-metrics-daemon-4r4zh" (UID: "8279f51a-80ab-4d6f-98b0-32805b4d0355") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 19 07:42:18.181755 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:18.181641 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clglj\" (UniqueName: \"kubernetes.io/projected/8320ca86-7c1c-45ce-9fe9-ee6d779d5d4d-kube-api-access-clglj\") pod \"network-check-target-lnxd5\" (UID: \"8320ca86-7c1c-45ce-9fe9-ee6d779d5d4d\") " pod="openshift-network-diagnostics/network-check-target-lnxd5" Jul 19 07:42:18.181755 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:18.181698 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/17c9482e-aaa4-41b0-8aad-ac6f428f1f48-original-pull-secret\") pod \"global-pull-secret-syncer-p5wwk\" (UID: \"17c9482e-aaa4-41b0-8aad-ac6f428f1f48\") " pod="kube-system/global-pull-secret-syncer-p5wwk" Jul 19 07:42:18.181961 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:18.181887 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 19 07:42:18.181961 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:18.181957 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17c9482e-aaa4-41b0-8aad-ac6f428f1f48-original-pull-secret podName:17c9482e-aaa4-41b0-8aad-ac6f428f1f48 nodeName:}" failed. No retries permitted until 2026-07-19 07:42:20.181937133 +0000 UTC m=+6.536246735 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/17c9482e-aaa4-41b0-8aad-ac6f428f1f48-original-pull-secret") pod "global-pull-secret-syncer-p5wwk" (UID: "17c9482e-aaa4-41b0-8aad-ac6f428f1f48") : object "kube-system"/"original-pull-secret" not registered Jul 19 07:42:18.182395 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:18.182374 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 19 07:42:18.182481 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:18.182399 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 19 07:42:18.182481 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:18.182415 2574 projected.go:196] Error preparing data for projected volume kube-api-access-clglj for pod openshift-network-diagnostics/network-check-target-lnxd5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 19 07:42:18.182481 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:18.182471 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8320ca86-7c1c-45ce-9fe9-ee6d779d5d4d-kube-api-access-clglj podName:8320ca86-7c1c-45ce-9fe9-ee6d779d5d4d nodeName:}" failed. No retries permitted until 2026-07-19 07:42:20.182454717 +0000 UTC m=+6.536764321 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-clglj" (UniqueName: "kubernetes.io/projected/8320ca86-7c1c-45ce-9fe9-ee6d779d5d4d-kube-api-access-clglj") pod "network-check-target-lnxd5" (UID: "8320ca86-7c1c-45ce-9fe9-ee6d779d5d4d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 19 07:42:18.494428 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:18.494323 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-18 07:37:16 +0000 UTC" deadline="2028-02-14 23:55:30.682922356 +0000 UTC" Jul 19 07:42:18.494428 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:18.494395 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13816h13m12.188531466s" Jul 19 07:42:18.555308 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:18.555273 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4r4zh" Jul 19 07:42:18.555474 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:18.555428 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4r4zh" podUID="8279f51a-80ab-4d6f-98b0-32805b4d0355" Jul 19 07:42:18.579383 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:18.579257 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nrkg5" event={"ID":"3e07cf7b-66a0-4756-b8d6-dcaccc94dc52","Type":"ContainerStarted","Data":"ade52d958857296be0edd22b1e4a8631a59ec8b809e0a7005f3cb2ecbcfb1c93"} Jul 19 07:42:18.598162 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:18.598118 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-k76h2" event={"ID":"dba80f1e-6d8a-47c2-a0dc-034f93da275b","Type":"ContainerStarted","Data":"f3e9957abae13628ebec18a699e3abfea8fa027eab596c17ca5db725b4129078"} Jul 19 07:42:18.602044 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:18.602001 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-79f6n" event={"ID":"62c0ff0a-c82e-4d0e-a721-9a49d1d59e11","Type":"ContainerStarted","Data":"c7ce1fe57b7b7592c1a342b8cc45a5482fe37d08048746a58985f1bbda4d7e9c"} Jul 19 07:42:18.622574 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:18.622537 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wskg4" event={"ID":"c52de7f2-39fd-4f3f-ae50-ef961ed37926","Type":"ContainerStarted","Data":"e9541d57ab0cf254fe5fce5d94d673ad8ae3236f1fdd3825ca7d8c34461a9315"} Jul 19 07:42:18.636319 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:18.636276 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5dh7h" event={"ID":"cecbc684-8b2f-4c00-a7ac-2966f0d95873","Type":"ContainerStarted","Data":"fb7516c01f33f99d57a5d69a2b097df21f61c029c32ea3eb160bcf3922831008"} Jul 19 07:42:18.666618 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:18.666552 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-bnnwq" event={"ID":"8259d1a2-204c-4d43-a9ce-939a685dc44f","Type":"ContainerStarted","Data":"007558a4c66acbe4a4600fff4b42ba9ed8cec0ad09dd48f35c85081e317ca403"} Jul 19 07:42:18.679263 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:18.679225 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-f5pcm" event={"ID":"8cea20a0-fc13-42b5-8c65-a67697252f18","Type":"ContainerStarted","Data":"323c142f85f5692ed7c8269d0e351918a5fe244d9a237ec5c0bf4d5f1cb09c0a"} Jul 19 07:42:18.689101 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:18.688939 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ctt95" event={"ID":"67825d1c-90c9-4e7f-87c8-575b38afa13a","Type":"ContainerStarted","Data":"19ad32232894efe94059acd25fb8ad1ed692ecd73f883d7f6fd6c25cb681cec8"} Jul 19 07:42:18.703998 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:18.703913 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lpbgg" event={"ID":"6d024bd7-3f48-4487-b740-2bb4a4e197fc","Type":"ContainerStarted","Data":"ee363777e3de6b487fb1b10a0d8e59d8f8ad765c9c0ffc496e1e66839e941db3"} Jul 19 07:42:18.732038 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:18.731999 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-70.ec2.internal" event={"ID":"76b974fafc626c9019b0378af47d7689","Type":"ContainerStarted","Data":"7f39fef75fd7ee8476a2e26f9ecdf7dc24d057d109f82d5929e9e13a910129b6"} Jul 19 07:42:18.761181 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:18.758832 2574 generic.go:356] "Generic (PLEG): container finished" podID="2e9c98df0848b5c4bc3c830d475a22f3" containerID="71926d88988ab6abe9f920e224bbfb5497b2fd008234e3d4a42262864aafc282" exitCode=0 Jul 19 07:42:18.761181 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:18.758875 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-70.ec2.internal" event={"ID":"2e9c98df0848b5c4bc3c830d475a22f3","Type":"ContainerDied","Data":"71926d88988ab6abe9f920e224bbfb5497b2fd008234e3d4a42262864aafc282"} Jul 19 07:42:18.774843 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:18.774778 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-70.ec2.internal" podStartSLOduration=3.774736549 podStartE2EDuration="3.774736549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-19 07:42:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:42:18.747469095 +0000 UTC m=+5.101778697" watchObservedRunningTime="2026-07-19 07:42:18.774736549 +0000 UTC m=+5.129046151" Jul 19 07:42:19.552492 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:19.552458 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lnxd5" Jul 19 07:42:19.552964 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:19.552618 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lnxd5" podUID="8320ca86-7c1c-45ce-9fe9-ee6d779d5d4d" Jul 19 07:42:19.553117 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:19.553028 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p5wwk" Jul 19 07:42:19.553184 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:19.553128 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p5wwk" podUID="17c9482e-aaa4-41b0-8aad-ac6f428f1f48" Jul 19 07:42:19.781349 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:19.781320 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-70.ec2.internal" event={"ID":"2e9c98df0848b5c4bc3c830d475a22f3","Type":"ContainerStarted","Data":"e68abd679306919474d1f81d751f7fbef21cec8ca4dc9ac961ce2ef1857a3adf"} Jul 19 07:42:20.100046 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:20.100009 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8279f51a-80ab-4d6f-98b0-32805b4d0355-metrics-certs\") pod \"network-metrics-daemon-4r4zh\" (UID: \"8279f51a-80ab-4d6f-98b0-32805b4d0355\") " pod="openshift-multus/network-metrics-daemon-4r4zh" Jul 19 07:42:20.100227 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:20.100209 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 19 07:42:20.100312 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:20.100282 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8279f51a-80ab-4d6f-98b0-32805b4d0355-metrics-certs podName:8279f51a-80ab-4d6f-98b0-32805b4d0355 nodeName:}" failed. No retries permitted until 2026-07-19 07:42:24.100261751 +0000 UTC m=+10.454571346 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8279f51a-80ab-4d6f-98b0-32805b4d0355-metrics-certs") pod "network-metrics-daemon-4r4zh" (UID: "8279f51a-80ab-4d6f-98b0-32805b4d0355") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 19 07:42:20.201475 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:20.200596 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clglj\" (UniqueName: \"kubernetes.io/projected/8320ca86-7c1c-45ce-9fe9-ee6d779d5d4d-kube-api-access-clglj\") pod \"network-check-target-lnxd5\" (UID: \"8320ca86-7c1c-45ce-9fe9-ee6d779d5d4d\") " pod="openshift-network-diagnostics/network-check-target-lnxd5" Jul 19 07:42:20.201475 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:20.200654 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/17c9482e-aaa4-41b0-8aad-ac6f428f1f48-original-pull-secret\") pod \"global-pull-secret-syncer-p5wwk\" (UID: \"17c9482e-aaa4-41b0-8aad-ac6f428f1f48\") " pod="kube-system/global-pull-secret-syncer-p5wwk" Jul 19 07:42:20.201475 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:20.200831 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 19 07:42:20.201475 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:20.200902 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17c9482e-aaa4-41b0-8aad-ac6f428f1f48-original-pull-secret podName:17c9482e-aaa4-41b0-8aad-ac6f428f1f48 nodeName:}" failed. No retries permitted until 2026-07-19 07:42:24.200882742 +0000 UTC m=+10.555192344 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/17c9482e-aaa4-41b0-8aad-ac6f428f1f48-original-pull-secret") pod "global-pull-secret-syncer-p5wwk" (UID: "17c9482e-aaa4-41b0-8aad-ac6f428f1f48") : object "kube-system"/"original-pull-secret" not registered Jul 19 07:42:20.201475 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:20.201347 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 19 07:42:20.201475 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:20.201368 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 19 07:42:20.201475 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:20.201381 2574 projected.go:196] Error preparing data for projected volume kube-api-access-clglj for pod openshift-network-diagnostics/network-check-target-lnxd5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 19 07:42:20.201475 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:20.201431 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8320ca86-7c1c-45ce-9fe9-ee6d779d5d4d-kube-api-access-clglj podName:8320ca86-7c1c-45ce-9fe9-ee6d779d5d4d nodeName:}" failed. No retries permitted until 2026-07-19 07:42:24.201415798 +0000 UTC m=+10.555725378 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-clglj" (UniqueName: "kubernetes.io/projected/8320ca86-7c1c-45ce-9fe9-ee6d779d5d4d-kube-api-access-clglj") pod "network-check-target-lnxd5" (UID: "8320ca86-7c1c-45ce-9fe9-ee6d779d5d4d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 19 07:42:20.554963 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:20.554929 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4r4zh" Jul 19 07:42:20.555424 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:20.555086 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4r4zh" podUID="8279f51a-80ab-4d6f-98b0-32805b4d0355" Jul 19 07:42:21.553006 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:21.552504 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lnxd5" Jul 19 07:42:21.553006 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:21.552653 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lnxd5" podUID="8320ca86-7c1c-45ce-9fe9-ee6d779d5d4d" Jul 19 07:42:21.553006 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:21.552532 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p5wwk" Jul 19 07:42:21.553006 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:21.552783 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p5wwk" podUID="17c9482e-aaa4-41b0-8aad-ac6f428f1f48" Jul 19 07:42:22.553243 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:22.553218 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4r4zh" Jul 19 07:42:22.553651 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:22.553358 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4r4zh" podUID="8279f51a-80ab-4d6f-98b0-32805b4d0355" Jul 19 07:42:23.552856 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:23.552811 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p5wwk" Jul 19 07:42:23.552856 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:23.552845 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lnxd5" Jul 19 07:42:23.553113 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:23.552969 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p5wwk" podUID="17c9482e-aaa4-41b0-8aad-ac6f428f1f48" Jul 19 07:42:23.553113 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:23.553086 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lnxd5" podUID="8320ca86-7c1c-45ce-9fe9-ee6d779d5d4d" Jul 19 07:42:24.135307 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:24.135137 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8279f51a-80ab-4d6f-98b0-32805b4d0355-metrics-certs\") pod \"network-metrics-daemon-4r4zh\" (UID: \"8279f51a-80ab-4d6f-98b0-32805b4d0355\") " pod="openshift-multus/network-metrics-daemon-4r4zh" Jul 19 07:42:24.135850 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:24.135320 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 19 07:42:24.135850 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:24.135390 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8279f51a-80ab-4d6f-98b0-32805b4d0355-metrics-certs podName:8279f51a-80ab-4d6f-98b0-32805b4d0355 nodeName:}" failed. No retries permitted until 2026-07-19 07:42:32.135370371 +0000 UTC m=+18.489679952 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8279f51a-80ab-4d6f-98b0-32805b4d0355-metrics-certs") pod "network-metrics-daemon-4r4zh" (UID: "8279f51a-80ab-4d6f-98b0-32805b4d0355") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 19 07:42:24.236589 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:24.236552 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clglj\" (UniqueName: \"kubernetes.io/projected/8320ca86-7c1c-45ce-9fe9-ee6d779d5d4d-kube-api-access-clglj\") pod \"network-check-target-lnxd5\" (UID: \"8320ca86-7c1c-45ce-9fe9-ee6d779d5d4d\") " pod="openshift-network-diagnostics/network-check-target-lnxd5" Jul 19 07:42:24.236770 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:24.236614 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/17c9482e-aaa4-41b0-8aad-ac6f428f1f48-original-pull-secret\") pod \"global-pull-secret-syncer-p5wwk\" (UID: \"17c9482e-aaa4-41b0-8aad-ac6f428f1f48\") " pod="kube-system/global-pull-secret-syncer-p5wwk" Jul 19 07:42:24.236770 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:24.236736 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 19 07:42:24.236770 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:24.236762 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 19 07:42:24.236944 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:24.236777 2574 projected.go:196] Error preparing data for projected volume kube-api-access-clglj for pod openshift-network-diagnostics/network-check-target-lnxd5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 19 07:42:24.236944 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:24.236734 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 19 07:42:24.236944 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:24.236838 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8320ca86-7c1c-45ce-9fe9-ee6d779d5d4d-kube-api-access-clglj podName:8320ca86-7c1c-45ce-9fe9-ee6d779d5d4d nodeName:}" failed. No retries permitted until 2026-07-19 07:42:32.236818699 +0000 UTC m=+18.591128307 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-clglj" (UniqueName: "kubernetes.io/projected/8320ca86-7c1c-45ce-9fe9-ee6d779d5d4d-kube-api-access-clglj") pod "network-check-target-lnxd5" (UID: "8320ca86-7c1c-45ce-9fe9-ee6d779d5d4d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 19 07:42:24.236944 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:24.236922 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17c9482e-aaa4-41b0-8aad-ac6f428f1f48-original-pull-secret podName:17c9482e-aaa4-41b0-8aad-ac6f428f1f48 nodeName:}" failed. No retries permitted until 2026-07-19 07:42:32.236902156 +0000 UTC m=+18.591211740 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/17c9482e-aaa4-41b0-8aad-ac6f428f1f48-original-pull-secret") pod "global-pull-secret-syncer-p5wwk" (UID: "17c9482e-aaa4-41b0-8aad-ac6f428f1f48") : object "kube-system"/"original-pull-secret" not registered Jul 19 07:42:24.553535 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:24.553490 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4r4zh" Jul 19 07:42:24.553723 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:24.553627 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4r4zh" podUID="8279f51a-80ab-4d6f-98b0-32805b4d0355" Jul 19 07:42:25.552913 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:25.552879 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lnxd5" Jul 19 07:42:25.553377 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:25.552934 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p5wwk" Jul 19 07:42:25.553377 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:25.553075 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lnxd5" podUID="8320ca86-7c1c-45ce-9fe9-ee6d779d5d4d" Jul 19 07:42:25.553377 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:25.553161 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p5wwk" podUID="17c9482e-aaa4-41b0-8aad-ac6f428f1f48" Jul 19 07:42:26.552625 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:26.552586 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4r4zh" Jul 19 07:42:26.552810 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:26.552738 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4r4zh" podUID="8279f51a-80ab-4d6f-98b0-32805b4d0355" Jul 19 07:42:27.552625 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:27.552504 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lnxd5" Jul 19 07:42:27.553109 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:27.552662 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lnxd5" podUID="8320ca86-7c1c-45ce-9fe9-ee6d779d5d4d" Jul 19 07:42:27.553109 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:27.552512 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p5wwk" Jul 19 07:42:27.553109 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:27.552814 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p5wwk" podUID="17c9482e-aaa4-41b0-8aad-ac6f428f1f48" Jul 19 07:42:28.552686 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:28.552605 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4r4zh" Jul 19 07:42:28.553065 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:28.552760 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4r4zh" podUID="8279f51a-80ab-4d6f-98b0-32805b4d0355" Jul 19 07:42:29.552878 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:29.552843 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lnxd5" Jul 19 07:42:29.553301 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:29.552854 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p5wwk" Jul 19 07:42:29.553301 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:29.552976 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lnxd5" podUID="8320ca86-7c1c-45ce-9fe9-ee6d779d5d4d" Jul 19 07:42:29.553301 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:29.553092 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p5wwk" podUID="17c9482e-aaa4-41b0-8aad-ac6f428f1f48" Jul 19 07:42:30.552609 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:30.552573 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4r4zh" Jul 19 07:42:30.552773 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:30.552731 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4r4zh" podUID="8279f51a-80ab-4d6f-98b0-32805b4d0355" Jul 19 07:42:31.552141 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:31.552100 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lnxd5" Jul 19 07:42:31.552684 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:31.552235 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lnxd5" podUID="8320ca86-7c1c-45ce-9fe9-ee6d779d5d4d" Jul 19 07:42:31.552684 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:31.552104 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p5wwk" Jul 19 07:42:31.552684 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:31.552392 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p5wwk" podUID="17c9482e-aaa4-41b0-8aad-ac6f428f1f48" Jul 19 07:42:32.197970 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:32.197932 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8279f51a-80ab-4d6f-98b0-32805b4d0355-metrics-certs\") pod \"network-metrics-daemon-4r4zh\" (UID: \"8279f51a-80ab-4d6f-98b0-32805b4d0355\") " pod="openshift-multus/network-metrics-daemon-4r4zh" Jul 19 07:42:32.198165 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:32.198105 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 19 07:42:32.198243 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:32.198186 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8279f51a-80ab-4d6f-98b0-32805b4d0355-metrics-certs podName:8279f51a-80ab-4d6f-98b0-32805b4d0355 nodeName:}" failed. No retries permitted until 2026-07-19 07:42:48.198169476 +0000 UTC m=+34.552479055 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8279f51a-80ab-4d6f-98b0-32805b4d0355-metrics-certs") pod "network-metrics-daemon-4r4zh" (UID: "8279f51a-80ab-4d6f-98b0-32805b4d0355") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 19 07:42:32.298621 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:32.298570 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clglj\" (UniqueName: \"kubernetes.io/projected/8320ca86-7c1c-45ce-9fe9-ee6d779d5d4d-kube-api-access-clglj\") pod \"network-check-target-lnxd5\" (UID: \"8320ca86-7c1c-45ce-9fe9-ee6d779d5d4d\") " pod="openshift-network-diagnostics/network-check-target-lnxd5" Jul 19 07:42:32.298772 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:32.298636 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/17c9482e-aaa4-41b0-8aad-ac6f428f1f48-original-pull-secret\") pod \"global-pull-secret-syncer-p5wwk\" (UID: \"17c9482e-aaa4-41b0-8aad-ac6f428f1f48\") " pod="kube-system/global-pull-secret-syncer-p5wwk" Jul 19 07:42:32.298818 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:32.298762 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 19 07:42:32.298862 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:32.298837 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17c9482e-aaa4-41b0-8aad-ac6f428f1f48-original-pull-secret podName:17c9482e-aaa4-41b0-8aad-ac6f428f1f48 nodeName:}" failed. No retries permitted until 2026-07-19 07:42:48.298822399 +0000 UTC m=+34.653131979 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/17c9482e-aaa4-41b0-8aad-ac6f428f1f48-original-pull-secret") pod "global-pull-secret-syncer-p5wwk" (UID: "17c9482e-aaa4-41b0-8aad-ac6f428f1f48") : object "kube-system"/"original-pull-secret" not registered Jul 19 07:42:32.298921 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:32.298856 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 19 07:42:32.298921 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:32.298884 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 19 07:42:32.298921 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:32.298901 2574 projected.go:196] Error preparing data for projected volume kube-api-access-clglj for pod openshift-network-diagnostics/network-check-target-lnxd5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 19 07:42:32.299007 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:32.298958 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8320ca86-7c1c-45ce-9fe9-ee6d779d5d4d-kube-api-access-clglj podName:8320ca86-7c1c-45ce-9fe9-ee6d779d5d4d nodeName:}" failed. No retries permitted until 2026-07-19 07:42:48.298942332 +0000 UTC m=+34.653251929 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-clglj" (UniqueName: "kubernetes.io/projected/8320ca86-7c1c-45ce-9fe9-ee6d779d5d4d-kube-api-access-clglj") pod "network-check-target-lnxd5" (UID: "8320ca86-7c1c-45ce-9fe9-ee6d779d5d4d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 19 07:42:32.553046 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:32.553012 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4r4zh" Jul 19 07:42:32.553410 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:32.553136 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4r4zh" podUID="8279f51a-80ab-4d6f-98b0-32805b4d0355" Jul 19 07:42:33.552897 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:33.552855 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lnxd5" Jul 19 07:42:33.553072 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:33.552866 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p5wwk" Jul 19 07:42:33.553072 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:33.552993 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lnxd5" podUID="8320ca86-7c1c-45ce-9fe9-ee6d779d5d4d" Jul 19 07:42:33.553466 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:33.553081 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p5wwk" podUID="17c9482e-aaa4-41b0-8aad-ac6f428f1f48" Jul 19 07:42:34.553424 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:34.553387 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4r4zh" Jul 19 07:42:34.553880 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:34.553536 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4r4zh" podUID="8279f51a-80ab-4d6f-98b0-32805b4d0355" Jul 19 07:42:35.553032 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:35.552989 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p5wwk" Jul 19 07:42:35.553032 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:35.553012 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lnxd5" Jul 19 07:42:35.553252 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:35.553141 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p5wwk" podUID="17c9482e-aaa4-41b0-8aad-ac6f428f1f48" Jul 19 07:42:35.553318 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:35.553259 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lnxd5" podUID="8320ca86-7c1c-45ce-9fe9-ee6d779d5d4d" Jul 19 07:42:36.552848 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:36.552564 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4r4zh" Jul 19 07:42:36.553129 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:36.552989 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4r4zh" podUID="8279f51a-80ab-4d6f-98b0-32805b4d0355" Jul 19 07:42:36.818246 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:36.818039 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-bnnwq" event={"ID":"8259d1a2-204c-4d43-a9ce-939a685dc44f","Type":"ContainerStarted","Data":"1e406db891702df0fac0c7eea365e21ac397e2bd3fd62bdeee4a03ee1e380503"} Jul 19 07:42:36.819418 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:36.819385 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ctt95" event={"ID":"67825d1c-90c9-4e7f-87c8-575b38afa13a","Type":"ContainerStarted","Data":"f433464ee586af9ef62d482dab3a7c5ab9dba564bf8cb0b722fb96d230693429"} Jul 19 07:42:36.820605 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:36.820580 2574 generic.go:356] "Generic (PLEG): container finished" podID="6d024bd7-3f48-4487-b740-2bb4a4e197fc" containerID="1405acb4370375188630d6c59c73c27eefa26e6b1d513bd5948c32160e33dfe5" exitCode=0 Jul 19 07:42:36.820702 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:36.820623 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lpbgg" event={"ID":"6d024bd7-3f48-4487-b740-2bb4a4e197fc","Type":"ContainerDied","Data":"1405acb4370375188630d6c59c73c27eefa26e6b1d513bd5948c32160e33dfe5"} Jul 19 07:42:36.823005 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:36.822985 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nrkg5" event={"ID":"3e07cf7b-66a0-4756-b8d6-dcaccc94dc52","Type":"ContainerStarted","Data":"6736d873e95c2bf8725e7ba38bc9140b2297d7524ea2edefeac1523a3f5b9400"} Jul 19 07:42:36.823101 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:36.823062 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nrkg5" event={"ID":"3e07cf7b-66a0-4756-b8d6-dcaccc94dc52","Type":"ContainerStarted","Data":"929ffc1c11e5fb71f080c6a3be30d09c6b57c79cca9180dc2cf64c1842f30439"} Jul 19 07:42:36.823169 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:36.823111 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nrkg5" event={"ID":"3e07cf7b-66a0-4756-b8d6-dcaccc94dc52","Type":"ContainerStarted","Data":"ef4cbfe51a38f4543d3d7aca88136a745000519d4cb35ecb91d644db3e6e4c0d"} Jul 19 07:42:36.824705 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:36.824684 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-k76h2" event={"ID":"dba80f1e-6d8a-47c2-a0dc-034f93da275b","Type":"ContainerStarted","Data":"dc9d436a4ccd609d64bfc56e27e3c19b59013e7d405c6827ec27c99fefb69144"} Jul 19 07:42:36.826179 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:36.826153 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-79f6n" event={"ID":"62c0ff0a-c82e-4d0e-a721-9a49d1d59e11","Type":"ContainerStarted","Data":"2dac9540882f4b765092cfb299985967eabec5270b99eb87fc98cac6cadb7bf1"} Jul 19 07:42:36.828799 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:36.828737 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wskg4" event={"ID":"c52de7f2-39fd-4f3f-ae50-ef961ed37926","Type":"ContainerStarted","Data":"2e590b3c605366615b0febab2d7aa9593997478e5b71a5fa18810be647734cc0"} Jul 19 07:42:36.830270 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:36.830253 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5dh7h" event={"ID":"cecbc684-8b2f-4c00-a7ac-2966f0d95873","Type":"ContainerStarted","Data":"2862ade7f44ba5e93569a020e14802d42a3eaf00e2439843ead46608ff71287c"} Jul 19 07:42:36.846540 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:36.846352 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-70.ec2.internal" podStartSLOduration=21.846332629 podStartE2EDuration="21.846332629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-19 07:42:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:42:19.796412097 +0000 UTC m=+6.150721700" watchObservedRunningTime="2026-07-19 07:42:36.846332629 +0000 UTC m=+23.200642235" Jul 19 07:42:36.878294 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:36.878253 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-5dh7h" podStartSLOduration=4.121363411 podStartE2EDuration="22.878239187s" totalImagesPullingTime="18.756875776s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-19 07:42:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:42:36.878224497 +0000 UTC m=+23.232534098" watchObservedRunningTime="2026-07-19 07:42:36.878239187 +0000 UTC m=+23.232548788" Jul 19 07:42:36.878813 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:36.878786 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-k76h2" podStartSLOduration=4.02518152 podStartE2EDuration="22.878778172s" totalImagesPullingTime="18.853596652s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-19 07:42:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:42:36.864686704 +0000 UTC m=+23.218996306" watchObservedRunningTime="2026-07-19 07:42:36.878778172 +0000 UTC m=+23.233087773" Jul 19 07:42:36.890979 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:36.890940 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-ctt95" podStartSLOduration=4.132319769 podStartE2EDuration="22.890925943s" totalImagesPullingTime="18.758606174s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-19 07:42:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:42:36.890507865 +0000 UTC m=+23.244817467" watchObservedRunningTime="2026-07-19 07:42:36.890925943 +0000 UTC m=+23.245235545" Jul 19 07:42:36.907272 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:36.907229 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-79f6n" podStartSLOduration=4.118409796 podStartE2EDuration="22.907213791s" totalImagesPullingTime="18.788803995s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-19 07:42:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:42:36.906831709 +0000 UTC m=+23.261141311" watchObservedRunningTime="2026-07-19 07:42:36.907213791 +0000 UTC m=+23.261523392" Jul 19 07:42:36.925655 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:36.925613 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-bnnwq" podStartSLOduration=11.249151342 podStartE2EDuration="21.925596522s" totalImagesPullingTime="10.67644518s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 07:42:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:42:36.925268267 +0000 UTC m=+23.279577869" watchObservedRunningTime="2026-07-19 07:42:36.925596522 +0000 UTC m=+23.279906123" Jul 19 07:42:37.077260 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:37.077235 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-bnnwq" Jul 19 07:42:37.077738 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:37.077722 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-bnnwq" Jul 19 07:42:37.552997 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:37.552938 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lnxd5" Jul 19 07:42:37.553969 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:37.552950 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p5wwk" Jul 19 07:42:37.553969 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:37.553065 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lnxd5" podUID="8320ca86-7c1c-45ce-9fe9-ee6d779d5d4d" Jul 19 07:42:37.553969 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:37.553123 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p5wwk" podUID="17c9482e-aaa4-41b0-8aad-ac6f428f1f48" Jul 19 07:42:37.834956 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:37.834639 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-f5pcm" event={"ID":"8cea20a0-fc13-42b5-8c65-a67697252f18","Type":"ContainerStarted","Data":"7aa20ab59073de4804407469cf2d37de59a1055b1df1228527263539c6ff4064"} Jul 19 07:42:37.838295 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:37.838259 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nrkg5" event={"ID":"3e07cf7b-66a0-4756-b8d6-dcaccc94dc52","Type":"ContainerStarted","Data":"efff5265e14f824a13ee6870b6b7f16cd74ea06f009b2d2d88940f9a5bbb202b"} Jul 19 07:42:37.838446 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:37.838340 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nrkg5" event={"ID":"3e07cf7b-66a0-4756-b8d6-dcaccc94dc52","Type":"ContainerStarted","Data":"1dd1f68fa2b7b8b2cbf167e848a739c4a5bc545d963951dc11d77ef345b5e796"} Jul 19 07:42:37.838446 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:37.838379 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nrkg5" event={"ID":"3e07cf7b-66a0-4756-b8d6-dcaccc94dc52","Type":"ContainerStarted","Data":"df54951f521ec2e2da2b6dd1dd9b00bb4f697b15dea5569c3fab6e5862926e2c"} Jul 19 07:42:37.848125 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:37.848080 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-f5pcm" podStartSLOduration=5.089316506 podStartE2EDuration="23.848064862s" totalImagesPullingTime="18.758748356s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-19 07:42:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:42:37.847796115 +0000 UTC m=+24.202105717" watchObservedRunningTime="2026-07-19 07:42:37.848064862 +0000 UTC m=+24.202374464" Jul 19 07:42:38.113615 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:38.113570 2574 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 19 07:42:38.523560 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:38.523448 2574 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-19T07:42:38.113595974Z","UUID":"12bd8c68-7dd6-49a4-9ba5-3739622242af","Handler":null,"Name":"","Endpoint":""} Jul 19 07:42:38.524980 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:38.524961 2574 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 19 07:42:38.525092 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:38.524996 2574 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 19 07:42:38.552259 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:38.552232 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4r4zh" Jul 19 07:42:38.552435 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:38.552356 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4r4zh" podUID="8279f51a-80ab-4d6f-98b0-32805b4d0355" Jul 19 07:42:38.844018 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:38.843870 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wskg4" event={"ID":"c52de7f2-39fd-4f3f-ae50-ef961ed37926","Type":"ContainerStarted","Data":"5d618507ece68704b063cfa40522cc8352337a1e1ef8163d2e7a28960f23c7c5"} Jul 19 07:42:38.844018 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:38.843926 2574 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 19 07:42:39.552641 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:39.552608 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lnxd5" Jul 19 07:42:39.552836 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:39.552655 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p5wwk" Jul 19 07:42:39.552836 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:39.552732 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lnxd5" podUID="8320ca86-7c1c-45ce-9fe9-ee6d779d5d4d" Jul 19 07:42:39.553113 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:39.552930 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p5wwk" podUID="17c9482e-aaa4-41b0-8aad-ac6f428f1f48" Jul 19 07:42:39.850456 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:39.850143 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nrkg5" event={"ID":"3e07cf7b-66a0-4756-b8d6-dcaccc94dc52","Type":"ContainerStarted","Data":"d3ae43c8f8f2ce4e9049f4603e313507243bed51353dfac5737a6366f49e3c25"} Jul 19 07:42:39.852418 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:39.852390 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wskg4" event={"ID":"c52de7f2-39fd-4f3f-ae50-ef961ed37926","Type":"ContainerStarted","Data":"e48134a7511122bf624a12a11cd806fc5f6984126a1ec1bae219da5b0e42b15a"} Jul 19 07:42:39.869604 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:39.869552 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wskg4" podStartSLOduration=4.362787325 podStartE2EDuration="25.869537461s" totalImagesPullingTime="21.506750136s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-19 07:42:14 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:42:39.8691044 +0000 UTC m=+26.223414002" watchObservedRunningTime="2026-07-19 07:42:39.869537461 +0000 UTC m=+26.223847061" Jul 19 07:42:40.552576 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:40.552539 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4r4zh" Jul 19 07:42:40.552809 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:40.552687 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4r4zh" podUID="8279f51a-80ab-4d6f-98b0-32805b4d0355" Jul 19 07:42:41.552966 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:41.552921 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p5wwk" Jul 19 07:42:41.553639 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:41.552923 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lnxd5" Jul 19 07:42:41.553639 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:41.553050 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p5wwk" podUID="17c9482e-aaa4-41b0-8aad-ac6f428f1f48" Jul 19 07:42:41.553639 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:41.553130 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lnxd5" podUID="8320ca86-7c1c-45ce-9fe9-ee6d779d5d4d" Jul 19 07:42:42.312856 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:42.312810 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-bnnwq" Jul 19 07:42:42.313070 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:42.313013 2574 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 19 07:42:42.313537 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:42.313488 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-bnnwq" Jul 19 07:42:42.553130 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:42.553096 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4r4zh" Jul 19 07:42:42.553625 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:42.553223 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4r4zh" podUID="8279f51a-80ab-4d6f-98b0-32805b4d0355" Jul 19 07:42:42.863494 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:42.863308 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nrkg5" event={"ID":"3e07cf7b-66a0-4756-b8d6-dcaccc94dc52","Type":"ContainerStarted","Data":"819b3ddb5a5c2b3922d6b55d1f9b424451a03456758bc1de25807241cd9710d4"} Jul 19 07:42:42.888632 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:42.888590 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-nrkg5" podStartSLOduration=10.064376918 podStartE2EDuration="28.8885734s" totalImagesPullingTime="18.824196482s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-19 07:42:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:42:42.888034101 +0000 UTC m=+29.242343701" watchObservedRunningTime="2026-07-19 07:42:42.8885734 +0000 UTC m=+29.242883000" Jul 19 07:42:43.552186 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:43.552155 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p5wwk" Jul 19 07:42:43.552362 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:43.552155 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lnxd5" Jul 19 07:42:43.552362 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:43.552265 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p5wwk" podUID="17c9482e-aaa4-41b0-8aad-ac6f428f1f48" Jul 19 07:42:43.552362 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:43.552337 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lnxd5" podUID="8320ca86-7c1c-45ce-9fe9-ee6d779d5d4d" Jul 19 07:42:43.867894 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:43.867795 2574 generic.go:356] "Generic (PLEG): container finished" podID="6d024bd7-3f48-4487-b740-2bb4a4e197fc" containerID="58f253aec74a0e8aa20b48af2189ccaa1e5748bf55b412593ab08ec3dfbdd265" exitCode=0 Jul 19 07:42:43.869665 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:43.869627 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lpbgg" event={"ID":"6d024bd7-3f48-4487-b740-2bb4a4e197fc","Type":"ContainerDied","Data":"58f253aec74a0e8aa20b48af2189ccaa1e5748bf55b412593ab08ec3dfbdd265"} Jul 19 07:42:43.869788 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:43.869686 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nrkg5" Jul 19 07:42:43.869850 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:43.869816 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nrkg5" Jul 19 07:42:43.869899 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:43.869871 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nrkg5" Jul 19 07:42:43.891342 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:43.891299 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nrkg5" Jul 19 07:42:43.893102 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:43.893081 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nrkg5" Jul 19 07:42:44.553163 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:44.553109 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4r4zh" Jul 19 07:42:44.553468 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:44.553229 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4r4zh" podUID="8279f51a-80ab-4d6f-98b0-32805b4d0355" Jul 19 07:42:44.874499 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:44.874407 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-p5wwk"] Jul 19 07:42:44.874989 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:44.874542 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p5wwk" Jul 19 07:42:44.874989 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:44.874654 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p5wwk" podUID="17c9482e-aaa4-41b0-8aad-ac6f428f1f48" Jul 19 07:42:44.877691 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:44.877647 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-lnxd5"] Jul 19 07:42:44.877825 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:44.877776 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lnxd5" Jul 19 07:42:44.877883 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:44.877867 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lnxd5" podUID="8320ca86-7c1c-45ce-9fe9-ee6d779d5d4d" Jul 19 07:42:44.878282 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:44.878259 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4r4zh"] Jul 19 07:42:44.878401 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:44.878351 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4r4zh" Jul 19 07:42:44.878478 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:44.878457 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4r4zh" podUID="8279f51a-80ab-4d6f-98b0-32805b4d0355" Jul 19 07:42:45.878972 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:45.878201 2574 generic.go:356] "Generic (PLEG): container finished" podID="6d024bd7-3f48-4487-b740-2bb4a4e197fc" containerID="2a188fb865160eca0e4ce92101f7dc5331c0ef965c6e78f1e65332238a5412b4" exitCode=0 Jul 19 07:42:45.878972 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:45.878287 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lpbgg" event={"ID":"6d024bd7-3f48-4487-b740-2bb4a4e197fc","Type":"ContainerDied","Data":"2a188fb865160eca0e4ce92101f7dc5331c0ef965c6e78f1e65332238a5412b4"} Jul 19 07:42:46.552616 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:46.552576 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4r4zh" Jul 19 07:42:46.552616 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:46.552611 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lnxd5" Jul 19 07:42:46.552860 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:46.552730 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4r4zh" podUID="8279f51a-80ab-4d6f-98b0-32805b4d0355" Jul 19 07:42:46.552860 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:46.552816 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lnxd5" podUID="8320ca86-7c1c-45ce-9fe9-ee6d779d5d4d" Jul 19 07:42:46.552950 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:46.552875 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p5wwk" Jul 19 07:42:46.552989 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:46.552949 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p5wwk" podUID="17c9482e-aaa4-41b0-8aad-ac6f428f1f48" Jul 19 07:42:47.885961 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:47.885922 2574 generic.go:356] "Generic (PLEG): container finished" podID="6d024bd7-3f48-4487-b740-2bb4a4e197fc" containerID="b372973c8ecc9320b3fc750f54c5484a9b978816ce580430c11c92e1b59c212c" exitCode=0 Jul 19 07:42:47.886580 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:47.885978 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lpbgg" event={"ID":"6d024bd7-3f48-4487-b740-2bb4a4e197fc","Type":"ContainerDied","Data":"b372973c8ecc9320b3fc750f54c5484a9b978816ce580430c11c92e1b59c212c"} Jul 19 07:42:48.213597 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:48.213566 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8279f51a-80ab-4d6f-98b0-32805b4d0355-metrics-certs\") pod \"network-metrics-daemon-4r4zh\" (UID: \"8279f51a-80ab-4d6f-98b0-32805b4d0355\") " pod="openshift-multus/network-metrics-daemon-4r4zh" Jul 19 07:42:48.213777 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:48.213678 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 19 07:42:48.213777 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:48.213745 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8279f51a-80ab-4d6f-98b0-32805b4d0355-metrics-certs podName:8279f51a-80ab-4d6f-98b0-32805b4d0355 nodeName:}" failed. No retries permitted until 2026-07-19 07:43:20.213726355 +0000 UTC m=+66.568035948 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8279f51a-80ab-4d6f-98b0-32805b4d0355-metrics-certs") pod "network-metrics-daemon-4r4zh" (UID: "8279f51a-80ab-4d6f-98b0-32805b4d0355") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 19 07:42:48.314853 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:48.314819 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clglj\" (UniqueName: \"kubernetes.io/projected/8320ca86-7c1c-45ce-9fe9-ee6d779d5d4d-kube-api-access-clglj\") pod \"network-check-target-lnxd5\" (UID: \"8320ca86-7c1c-45ce-9fe9-ee6d779d5d4d\") " pod="openshift-network-diagnostics/network-check-target-lnxd5" Jul 19 07:42:48.315029 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:48.314871 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/17c9482e-aaa4-41b0-8aad-ac6f428f1f48-original-pull-secret\") pod \"global-pull-secret-syncer-p5wwk\" (UID: \"17c9482e-aaa4-41b0-8aad-ac6f428f1f48\") " pod="kube-system/global-pull-secret-syncer-p5wwk" Jul 19 07:42:48.315029 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:48.314972 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 19 07:42:48.315029 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:48.314980 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 19 07:42:48.315029 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:48.315003 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 19 07:42:48.315029 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:48.315016 2574 projected.go:196] Error preparing data for projected volume kube-api-access-clglj for pod openshift-network-diagnostics/network-check-target-lnxd5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 19 07:42:48.315029 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:48.315033 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17c9482e-aaa4-41b0-8aad-ac6f428f1f48-original-pull-secret podName:17c9482e-aaa4-41b0-8aad-ac6f428f1f48 nodeName:}" failed. No retries permitted until 2026-07-19 07:43:20.315020665 +0000 UTC m=+66.669330244 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/17c9482e-aaa4-41b0-8aad-ac6f428f1f48-original-pull-secret") pod "global-pull-secret-syncer-p5wwk" (UID: "17c9482e-aaa4-41b0-8aad-ac6f428f1f48") : object "kube-system"/"original-pull-secret" not registered Jul 19 07:42:48.315232 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:48.315062 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8320ca86-7c1c-45ce-9fe9-ee6d779d5d4d-kube-api-access-clglj podName:8320ca86-7c1c-45ce-9fe9-ee6d779d5d4d nodeName:}" failed. No retries permitted until 2026-07-19 07:43:20.315048278 +0000 UTC m=+66.669357877 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-clglj" (UniqueName: "kubernetes.io/projected/8320ca86-7c1c-45ce-9fe9-ee6d779d5d4d-kube-api-access-clglj") pod "network-check-target-lnxd5" (UID: "8320ca86-7c1c-45ce-9fe9-ee6d779d5d4d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 19 07:42:48.553064 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:48.552981 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p5wwk" Jul 19 07:42:48.553227 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:48.552984 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lnxd5" Jul 19 07:42:48.553227 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:48.553121 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p5wwk" podUID="17c9482e-aaa4-41b0-8aad-ac6f428f1f48" Jul 19 07:42:48.553227 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:48.552992 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4r4zh" Jul 19 07:42:48.553227 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:48.553184 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lnxd5" podUID="8320ca86-7c1c-45ce-9fe9-ee6d779d5d4d" Jul 19 07:42:48.553484 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:48.553270 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4r4zh" podUID="8279f51a-80ab-4d6f-98b0-32805b4d0355" Jul 19 07:42:50.349262 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.349230 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-70.ec2.internal" event="NodeReady" Jul 19 07:42:50.349718 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.349363 2574 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 19 07:42:50.381380 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.381348 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-2hx2z"] Jul 19 07:42:50.385199 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.385068 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-rtvvb"] Jul 19 07:42:50.385199 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.385194 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-2hx2z" Jul 19 07:42:50.387741 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.387719 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 19 07:42:50.387980 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.387962 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 19 07:42:50.388384 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.388369 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 19 07:42:50.388706 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.388693 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 19 07:42:50.388914 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.388900 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-snh6v\"" Jul 19 07:42:50.389188 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.389170 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-m5q94"] Jul 19 07:42:50.389277 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.389265 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-rtvvb" Jul 19 07:42:50.391728 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.391707 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 19 07:42:50.391811 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.391715 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-fbznh\"" Jul 19 07:42:50.391956 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.391943 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 19 07:42:50.393023 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.393002 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7dbdbbf55d-6mrcs"] Jul 19 07:42:50.393306 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.393291 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-m5q94" Jul 19 07:42:50.395619 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.395599 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 19 07:42:50.395705 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.395616 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-lz2qf\"" Jul 19 07:42:50.395705 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.395656 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 19 07:42:50.395705 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.395603 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 19 07:42:50.396825 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.396798 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-wmhfd"] Jul 19 07:42:50.396915 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.396892 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7dbdbbf55d-6mrcs" Jul 19 07:42:50.399001 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.398918 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-thnlm\"" Jul 19 07:42:50.399119 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.399040 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 19 07:42:50.399230 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.399170 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 19 07:42:50.399273 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.399239 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 19 07:42:50.404292 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.403360 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-555d9d7885-xr7x7"] Jul 19 07:42:50.404656 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.404631 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-565b7b569d-wmhfd" Jul 19 07:42:50.407379 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.407304 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-rtvvb"] Jul 19 07:42:50.407379 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.407350 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-2hx2z"] Jul 19 07:42:50.407379 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.407372 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-m5q94"] Jul 19 07:42:50.407611 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.407538 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-555d9d7885-xr7x7" Jul 19 07:42:50.407611 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.407575 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 19 07:42:50.409106 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.409031 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 19 07:42:50.409370 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.409315 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-bxkd6\"" Jul 19 07:42:50.412372 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.411571 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-l7s85"] Jul 19 07:42:50.415298 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.415261 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 19 07:42:50.416474 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.416443 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-tlm7l\"" Jul 19 07:42:50.416593 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.416513 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 19 07:42:50.417567 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.417421 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 19 07:42:50.417567 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.417500 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 19 07:42:50.418462 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.417969 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 19 07:42:50.418462 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.418434 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 19 07:42:50.419759 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.419737 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-5669bd84dd-j7hs7"] Jul 19 07:42:50.421216 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.421190 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 19 07:42:50.422355 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.422337 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 19 07:42:50.423884 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.423859 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-l7s85" Jul 19 07:42:50.427719 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.427700 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vcblr\"" Jul 19 07:42:50.427950 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.427771 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 19 07:42:50.428072 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.427821 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 19 07:42:50.428179 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.428001 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-hsd5x"] Jul 19 07:42:50.428368 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.428088 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5669bd84dd-j7hs7" Jul 19 07:42:50.431083 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.431065 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 19 07:42:50.431672 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.431644 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 19 07:42:50.432199 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.431956 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-5sgkp"] Jul 19 07:42:50.432592 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.432018 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-hsd5x" Jul 19 07:42:50.434023 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.433999 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 19 07:42:50.434117 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.434054 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 19 07:42:50.434586 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.434546 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 19 07:42:50.434901 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.434884 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 19 07:42:50.435116 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.434949 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 19 07:42:50.435315 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.435128 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-cqf77\"" Jul 19 07:42:50.436648 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.436625 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 19 07:42:50.436945 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.436925 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 19 07:42:50.437754 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.437734 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 19 07:42:50.438009 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.437992 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 19 07:42:50.439009 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.438974 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-8w7nz"] Jul 19 07:42:50.440172 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.440149 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-5sgkp" Jul 19 07:42:50.440611 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.440167 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-4v7ts\"" Jul 19 07:42:50.442511 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.442489 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 19 07:42:50.442770 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.442752 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 19 07:42:50.443275 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.443254 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-2r5vj\"" Jul 19 07:42:50.443442 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.443419 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 19 07:42:50.443733 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.443602 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 19 07:42:50.444472 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.444452 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-wmhfd"] Jul 19 07:42:50.444571 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.444492 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-m78pm"] Jul 19 07:42:50.444638 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.444582 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-8w7nz" Jul 19 07:42:50.447065 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.447044 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 19 07:42:50.447491 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.447328 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-pcq4x\"" Jul 19 07:42:50.447491 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.447385 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 19 07:42:50.448240 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.448209 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-x2qps"] Jul 19 07:42:50.448356 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.448337 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-m78pm" Jul 19 07:42:50.450838 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.450813 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-qmw7p\"" Jul 19 07:42:50.450937 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.450813 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 19 07:42:50.451001 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.450815 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 19 07:42:50.451051 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.450821 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 19 07:42:50.451740 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.451683 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7dbdbbf55d-6mrcs"] Jul 19 07:42:50.451740 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.451718 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-555d9d7885-xr7x7"] Jul 19 07:42:50.451740 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.451738 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-m78pm"] Jul 19 07:42:50.451970 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.451758 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-l7s85"] Jul 19 07:42:50.451970 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.451782 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-x2qps"] Jul 19 07:42:50.451970 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.451808 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5669bd84dd-j7hs7"] Jul 19 07:42:50.451970 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.451827 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-5sgkp"] Jul 19 07:42:50.451970 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.451846 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-hsd5x"] Jul 19 07:42:50.451970 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.451864 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-8w7nz"] Jul 19 07:42:50.451970 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.451866 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-x2qps" Jul 19 07:42:50.454119 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.454065 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 19 07:42:50.454119 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.454078 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 19 07:42:50.454291 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.454197 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-wmd7q\"" Jul 19 07:42:50.533161 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.533110 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt9tz\" (UniqueName: \"kubernetes.io/projected/d562d83b-2730-4701-b48b-43b8dc8651cc-kube-api-access-nt9tz\") pod \"volume-data-source-validator-78cf4c6f6-rtvvb\" (UID: \"d562d83b-2730-4701-b48b-43b8dc8651cc\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-rtvvb" Jul 19 07:42:50.533161 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.533156 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/f56affaf-b38d-4ea4-ac8c-309edaa8852e-default-certificate\") pod \"router-default-5669bd84dd-j7hs7\" (UID: \"f56affaf-b38d-4ea4-ac8c-309edaa8852e\") " pod="openshift-ingress/router-default-5669bd84dd-j7hs7" Jul 19 07:42:50.533438 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.533181 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ntqn\" (UniqueName: \"kubernetes.io/projected/9d4e8a76-432d-4750-bb41-9d86bdb458b9-kube-api-access-5ntqn\") pod \"image-registry-7dbdbbf55d-6mrcs\" (UID: \"9d4e8a76-432d-4750-bb41-9d86bdb458b9\") " pod="openshift-image-registry/image-registry-7dbdbbf55d-6mrcs" Jul 19 07:42:50.533438 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.533245 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f56affaf-b38d-4ea4-ac8c-309edaa8852e-service-ca-bundle\") pod \"router-default-5669bd84dd-j7hs7\" (UID: \"f56affaf-b38d-4ea4-ac8c-309edaa8852e\") " pod="openshift-ingress/router-default-5669bd84dd-j7hs7" Jul 19 07:42:50.533438 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.533268 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9d4e8a76-432d-4750-bb41-9d86bdb458b9-registry-certificates\") pod \"image-registry-7dbdbbf55d-6mrcs\" (UID: \"9d4e8a76-432d-4750-bb41-9d86bdb458b9\") " pod="openshift-image-registry/image-registry-7dbdbbf55d-6mrcs" Jul 19 07:42:50.533438 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.533299 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0adaffaf-8ab7-4966-abd1-5bbd8a0ba882-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-xr7x7\" (UID: \"0adaffaf-8ab7-4966-abd1-5bbd8a0ba882\") " pod="openshift-insights/insights-operator-555d9d7885-xr7x7" Jul 19 07:42:50.533438 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.533327 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e990c17f-7361-430a-abe2-8fe27dc88fd6-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-m5q94\" (UID: \"e990c17f-7361-430a-abe2-8fe27dc88fd6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-m5q94" Jul 19 07:42:50.533438 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.533363 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9d4e8a76-432d-4750-bb41-9d86bdb458b9-bound-sa-token\") pod \"image-registry-7dbdbbf55d-6mrcs\" (UID: \"9d4e8a76-432d-4750-bb41-9d86bdb458b9\") " pod="openshift-image-registry/image-registry-7dbdbbf55d-6mrcs" Jul 19 07:42:50.533438 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.533406 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7789b\" (UniqueName: \"kubernetes.io/projected/e990c17f-7361-430a-abe2-8fe27dc88fd6-kube-api-access-7789b\") pod \"cluster-samples-operator-79b7c55f87-m5q94\" (UID: \"e990c17f-7361-430a-abe2-8fe27dc88fd6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-m5q94" Jul 19 07:42:50.533438 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.533432 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/f56affaf-b38d-4ea4-ac8c-309edaa8852e-stats-auth\") pod \"router-default-5669bd84dd-j7hs7\" (UID: \"f56affaf-b38d-4ea4-ac8c-309edaa8852e\") " pod="openshift-ingress/router-default-5669bd84dd-j7hs7" Jul 19 07:42:50.533855 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.533455 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7h46q\" (UniqueName: \"kubernetes.io/projected/0adaffaf-8ab7-4966-abd1-5bbd8a0ba882-kube-api-access-7h46q\") pod \"insights-operator-555d9d7885-xr7x7\" (UID: \"0adaffaf-8ab7-4966-abd1-5bbd8a0ba882\") " pod="openshift-insights/insights-operator-555d9d7885-xr7x7" Jul 19 07:42:50.533855 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.533481 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a4349d67-121d-44bb-9d73-711ee0465237-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-8w7nz\" (UID: \"a4349d67-121d-44bb-9d73-711ee0465237\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-8w7nz" Jul 19 07:42:50.533855 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.533510 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xzcd\" (UniqueName: \"kubernetes.io/projected/dd68d31d-f763-433a-867b-65b12eabac76-kube-api-access-4xzcd\") pod \"console-operator-565b7b569d-wmhfd\" (UID: \"dd68d31d-f763-433a-867b-65b12eabac76\") " pod="openshift-console-operator/console-operator-565b7b569d-wmhfd" Jul 19 07:42:50.533855 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.533556 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9d4e8a76-432d-4750-bb41-9d86bdb458b9-ca-trust-extracted\") pod \"image-registry-7dbdbbf55d-6mrcs\" (UID: \"9d4e8a76-432d-4750-bb41-9d86bdb458b9\") " pod="openshift-image-registry/image-registry-7dbdbbf55d-6mrcs" Jul 19 07:42:50.533855 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.533584 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0adaffaf-8ab7-4966-abd1-5bbd8a0ba882-tmp\") pod \"insights-operator-555d9d7885-xr7x7\" (UID: \"0adaffaf-8ab7-4966-abd1-5bbd8a0ba882\") " pod="openshift-insights/insights-operator-555d9d7885-xr7x7" Jul 19 07:42:50.533855 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.533614 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0adaffaf-8ab7-4966-abd1-5bbd8a0ba882-service-ca-bundle\") pod \"insights-operator-555d9d7885-xr7x7\" (UID: \"0adaffaf-8ab7-4966-abd1-5bbd8a0ba882\") " pod="openshift-insights/insights-operator-555d9d7885-xr7x7" Jul 19 07:42:50.533855 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.533637 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/01c43e18-77be-4ec4-88b3-6a5662c66917-serving-cert\") pod \"service-ca-operator-5bcbc94c78-hsd5x\" (UID: \"01c43e18-77be-4ec4-88b3-6a5662c66917\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-hsd5x" Jul 19 07:42:50.533855 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.533665 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fc2e627b-03e6-48c5-a476-00eaa505368a-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-2hx2z\" (UID: \"fc2e627b-03e6-48c5-a476-00eaa505368a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-2hx2z" Jul 19 07:42:50.533855 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.533691 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9d4e8a76-432d-4750-bb41-9d86bdb458b9-registry-tls\") pod \"image-registry-7dbdbbf55d-6mrcs\" (UID: \"9d4e8a76-432d-4750-bb41-9d86bdb458b9\") " pod="openshift-image-registry/image-registry-7dbdbbf55d-6mrcs" Jul 19 07:42:50.533855 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.533717 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4821058e-2da0-4086-8911-6b394fb44223-metrics-tls\") pod \"dns-default-l7s85\" (UID: \"4821058e-2da0-4086-8911-6b394fb44223\") " pod="openshift-dns/dns-default-l7s85" Jul 19 07:42:50.533855 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.533743 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4821058e-2da0-4086-8911-6b394fb44223-tmp-dir\") pod \"dns-default-l7s85\" (UID: \"4821058e-2da0-4086-8911-6b394fb44223\") " pod="openshift-dns/dns-default-l7s85" Jul 19 07:42:50.533855 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.533808 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxsm4\" (UniqueName: \"kubernetes.io/projected/01c43e18-77be-4ec4-88b3-6a5662c66917-kube-api-access-cxsm4\") pod \"service-ca-operator-5bcbc94c78-hsd5x\" (UID: \"01c43e18-77be-4ec4-88b3-6a5662c66917\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-hsd5x" Jul 19 07:42:50.534364 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.533858 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5tmt\" (UniqueName: \"kubernetes.io/projected/33646489-0313-4d23-96c3-4b32f50233af-kube-api-access-v5tmt\") pod \"ingress-canary-m78pm\" (UID: \"33646489-0313-4d23-96c3-4b32f50233af\") " pod="openshift-ingress-canary/ingress-canary-m78pm" Jul 19 07:42:50.534364 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.533883 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ls5bd\" (UniqueName: \"kubernetes.io/projected/76040636-5309-4d6c-a14c-ffdc89085c2d-kube-api-access-ls5bd\") pod \"network-check-source-7bc5b6bf85-x2qps\" (UID: \"76040636-5309-4d6c-a14c-ffdc89085c2d\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-x2qps" Jul 19 07:42:50.534364 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.533907 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9d4e8a76-432d-4750-bb41-9d86bdb458b9-image-registry-private-configuration\") pod \"image-registry-7dbdbbf55d-6mrcs\" (UID: \"9d4e8a76-432d-4750-bb41-9d86bdb458b9\") " pod="openshift-image-registry/image-registry-7dbdbbf55d-6mrcs" Jul 19 07:42:50.534364 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.533956 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4821058e-2da0-4086-8911-6b394fb44223-config-volume\") pod \"dns-default-l7s85\" (UID: \"4821058e-2da0-4086-8911-6b394fb44223\") " pod="openshift-dns/dns-default-l7s85" Jul 19 07:42:50.534364 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.534005 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dd68d31d-f763-433a-867b-65b12eabac76-serving-cert\") pod \"console-operator-565b7b569d-wmhfd\" (UID: \"dd68d31d-f763-433a-867b-65b12eabac76\") " pod="openshift-console-operator/console-operator-565b7b569d-wmhfd" Jul 19 07:42:50.534364 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.534029 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9d4e8a76-432d-4750-bb41-9d86bdb458b9-installation-pull-secrets\") pod \"image-registry-7dbdbbf55d-6mrcs\" (UID: \"9d4e8a76-432d-4750-bb41-9d86bdb458b9\") " pod="openshift-image-registry/image-registry-7dbdbbf55d-6mrcs" Jul 19 07:42:50.534364 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.534069 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/33646489-0313-4d23-96c3-4b32f50233af-cert\") pod \"ingress-canary-m78pm\" (UID: \"33646489-0313-4d23-96c3-4b32f50233af\") " pod="openshift-ingress-canary/ingress-canary-m78pm" Jul 19 07:42:50.534364 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.534102 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mt7jb\" (UniqueName: \"kubernetes.io/projected/4821058e-2da0-4086-8911-6b394fb44223-kube-api-access-mt7jb\") pod \"dns-default-l7s85\" (UID: \"4821058e-2da0-4086-8911-6b394fb44223\") " pod="openshift-dns/dns-default-l7s85" Jul 19 07:42:50.534364 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.534128 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/01c43e18-77be-4ec4-88b3-6a5662c66917-config\") pod \"service-ca-operator-5bcbc94c78-hsd5x\" (UID: \"01c43e18-77be-4ec4-88b3-6a5662c66917\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-hsd5x" Jul 19 07:42:50.534364 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.534163 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f56affaf-b38d-4ea4-ac8c-309edaa8852e-metrics-certs\") pod \"router-default-5669bd84dd-j7hs7\" (UID: \"f56affaf-b38d-4ea4-ac8c-309edaa8852e\") " pod="openshift-ingress/router-default-5669bd84dd-j7hs7" Jul 19 07:42:50.534364 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.534192 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8sjx\" (UniqueName: \"kubernetes.io/projected/fc2e627b-03e6-48c5-a476-00eaa505368a-kube-api-access-v8sjx\") pod \"kube-storage-version-migrator-operator-7658d9b749-2hx2z\" (UID: \"fc2e627b-03e6-48c5-a476-00eaa505368a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-2hx2z" Jul 19 07:42:50.534364 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.534217 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jk2q\" (UniqueName: \"kubernetes.io/projected/fbd767a8-d8bc-4f92-a449-46474f1ac52d-kube-api-access-2jk2q\") pod \"cluster-monitoring-operator-76cf8966f8-5sgkp\" (UID: \"fbd767a8-d8bc-4f92-a449-46474f1ac52d\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-5sgkp" Jul 19 07:42:50.534364 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.534261 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dd68d31d-f763-433a-867b-65b12eabac76-config\") pod \"console-operator-565b7b569d-wmhfd\" (UID: \"dd68d31d-f763-433a-867b-65b12eabac76\") " pod="openshift-console-operator/console-operator-565b7b569d-wmhfd" Jul 19 07:42:50.534364 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.534292 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fbd767a8-d8bc-4f92-a449-46474f1ac52d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-5sgkp\" (UID: \"fbd767a8-d8bc-4f92-a449-46474f1ac52d\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-5sgkp" Jul 19 07:42:50.534364 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.534326 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fc2e627b-03e6-48c5-a476-00eaa505368a-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-2hx2z\" (UID: \"fc2e627b-03e6-48c5-a476-00eaa505368a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-2hx2z" Jul 19 07:42:50.534955 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.534349 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/0adaffaf-8ab7-4966-abd1-5bbd8a0ba882-snapshots\") pod \"insights-operator-555d9d7885-xr7x7\" (UID: \"0adaffaf-8ab7-4966-abd1-5bbd8a0ba882\") " pod="openshift-insights/insights-operator-555d9d7885-xr7x7" Jul 19 07:42:50.534955 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.534377 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9d4e8a76-432d-4750-bb41-9d86bdb458b9-trusted-ca\") pod \"image-registry-7dbdbbf55d-6mrcs\" (UID: \"9d4e8a76-432d-4750-bb41-9d86bdb458b9\") " pod="openshift-image-registry/image-registry-7dbdbbf55d-6mrcs" Jul 19 07:42:50.534955 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.534403 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a4349d67-121d-44bb-9d73-711ee0465237-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-8w7nz\" (UID: \"a4349d67-121d-44bb-9d73-711ee0465237\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-8w7nz" Jul 19 07:42:50.534955 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.534459 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dd68d31d-f763-433a-867b-65b12eabac76-trusted-ca\") pod \"console-operator-565b7b569d-wmhfd\" (UID: \"dd68d31d-f763-433a-867b-65b12eabac76\") " pod="openshift-console-operator/console-operator-565b7b569d-wmhfd" Jul 19 07:42:50.534955 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.534484 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4cfq\" (UniqueName: \"kubernetes.io/projected/f56affaf-b38d-4ea4-ac8c-309edaa8852e-kube-api-access-t4cfq\") pod \"router-default-5669bd84dd-j7hs7\" (UID: \"f56affaf-b38d-4ea4-ac8c-309edaa8852e\") " pod="openshift-ingress/router-default-5669bd84dd-j7hs7" Jul 19 07:42:50.534955 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.534512 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0adaffaf-8ab7-4966-abd1-5bbd8a0ba882-serving-cert\") pod \"insights-operator-555d9d7885-xr7x7\" (UID: \"0adaffaf-8ab7-4966-abd1-5bbd8a0ba882\") " pod="openshift-insights/insights-operator-555d9d7885-xr7x7" Jul 19 07:42:50.534955 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.534561 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/fbd767a8-d8bc-4f92-a449-46474f1ac52d-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-5sgkp\" (UID: \"fbd767a8-d8bc-4f92-a449-46474f1ac52d\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-5sgkp" Jul 19 07:42:50.552333 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.552306 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p5wwk" Jul 19 07:42:50.552508 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.552348 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4r4zh" Jul 19 07:42:50.552508 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.552312 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lnxd5" Jul 19 07:42:50.554718 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.554693 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 19 07:42:50.554878 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.554761 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-rrpxw\"" Jul 19 07:42:50.554878 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.554700 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 19 07:42:50.555247 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.555092 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ptfjq\"" Jul 19 07:42:50.556019 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.555964 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:42:50.632235 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.632139 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:42:50.635955 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.635919 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dd68d31d-f763-433a-867b-65b12eabac76-config\") pod \"console-operator-565b7b569d-wmhfd\" (UID: \"dd68d31d-f763-433a-867b-65b12eabac76\") " pod="openshift-console-operator/console-operator-565b7b569d-wmhfd" Jul 19 07:42:50.636104 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.635968 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fbd767a8-d8bc-4f92-a449-46474f1ac52d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-5sgkp\" (UID: \"fbd767a8-d8bc-4f92-a449-46474f1ac52d\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-5sgkp" Jul 19 07:42:50.636104 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.636003 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fc2e627b-03e6-48c5-a476-00eaa505368a-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-2hx2z\" (UID: \"fc2e627b-03e6-48c5-a476-00eaa505368a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-2hx2z" Jul 19 07:42:50.636104 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.636025 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/0adaffaf-8ab7-4966-abd1-5bbd8a0ba882-snapshots\") pod \"insights-operator-555d9d7885-xr7x7\" (UID: \"0adaffaf-8ab7-4966-abd1-5bbd8a0ba882\") " pod="openshift-insights/insights-operator-555d9d7885-xr7x7" Jul 19 07:42:50.636310 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:50.636150 2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 19 07:42:50.636310 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.636196 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9d4e8a76-432d-4750-bb41-9d86bdb458b9-trusted-ca\") pod \"image-registry-7dbdbbf55d-6mrcs\" (UID: \"9d4e8a76-432d-4750-bb41-9d86bdb458b9\") " pod="openshift-image-registry/image-registry-7dbdbbf55d-6mrcs" Jul 19 07:42:50.636310 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:50.636239 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbd767a8-d8bc-4f92-a449-46474f1ac52d-cluster-monitoring-operator-tls podName:fbd767a8-d8bc-4f92-a449-46474f1ac52d nodeName:}" failed. No retries permitted until 2026-07-19 07:42:51.136217447 +0000 UTC m=+37.490527042 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fbd767a8-d8bc-4f92-a449-46474f1ac52d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-5sgkp" (UID: "fbd767a8-d8bc-4f92-a449-46474f1ac52d") : secret "cluster-monitoring-operator-tls" not found Jul 19 07:42:50.636310 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.636279 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a4349d67-121d-44bb-9d73-711ee0465237-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-8w7nz\" (UID: \"a4349d67-121d-44bb-9d73-711ee0465237\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-8w7nz" Jul 19 07:42:50.636497 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.636309 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dd68d31d-f763-433a-867b-65b12eabac76-trusted-ca\") pod \"console-operator-565b7b569d-wmhfd\" (UID: \"dd68d31d-f763-433a-867b-65b12eabac76\") " pod="openshift-console-operator/console-operator-565b7b569d-wmhfd" Jul 19 07:42:50.636497 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.636337 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4cfq\" (UniqueName: \"kubernetes.io/projected/f56affaf-b38d-4ea4-ac8c-309edaa8852e-kube-api-access-t4cfq\") pod \"router-default-5669bd84dd-j7hs7\" (UID: \"f56affaf-b38d-4ea4-ac8c-309edaa8852e\") " pod="openshift-ingress/router-default-5669bd84dd-j7hs7" Jul 19 07:42:50.636497 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.636366 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0adaffaf-8ab7-4966-abd1-5bbd8a0ba882-serving-cert\") pod \"insights-operator-555d9d7885-xr7x7\" (UID: \"0adaffaf-8ab7-4966-abd1-5bbd8a0ba882\") " pod="openshift-insights/insights-operator-555d9d7885-xr7x7" Jul 19 07:42:50.636497 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.636396 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/fbd767a8-d8bc-4f92-a449-46474f1ac52d-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-5sgkp\" (UID: \"fbd767a8-d8bc-4f92-a449-46474f1ac52d\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-5sgkp" Jul 19 07:42:50.636497 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.636422 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt9tz\" (UniqueName: \"kubernetes.io/projected/d562d83b-2730-4701-b48b-43b8dc8651cc-kube-api-access-nt9tz\") pod \"volume-data-source-validator-78cf4c6f6-rtvvb\" (UID: \"d562d83b-2730-4701-b48b-43b8dc8651cc\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-rtvvb" Jul 19 07:42:50.636497 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:50.636430 2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 19 07:42:50.636497 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.636447 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/f56affaf-b38d-4ea4-ac8c-309edaa8852e-default-certificate\") pod \"router-default-5669bd84dd-j7hs7\" (UID: \"f56affaf-b38d-4ea4-ac8c-309edaa8852e\") " pod="openshift-ingress/router-default-5669bd84dd-j7hs7" Jul 19 07:42:50.636497 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.636470 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ntqn\" (UniqueName: \"kubernetes.io/projected/9d4e8a76-432d-4750-bb41-9d86bdb458b9-kube-api-access-5ntqn\") pod \"image-registry-7dbdbbf55d-6mrcs\" (UID: \"9d4e8a76-432d-4750-bb41-9d86bdb458b9\") " pod="openshift-image-registry/image-registry-7dbdbbf55d-6mrcs" Jul 19 07:42:50.636856 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:50.636507 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4349d67-121d-44bb-9d73-711ee0465237-networking-console-plugin-cert podName:a4349d67-121d-44bb-9d73-711ee0465237 nodeName:}" failed. No retries permitted until 2026-07-19 07:42:51.136488035 +0000 UTC m=+37.490797627 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a4349d67-121d-44bb-9d73-711ee0465237-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-8w7nz" (UID: "a4349d67-121d-44bb-9d73-711ee0465237") : secret "networking-console-plugin-cert" not found Jul 19 07:42:50.636856 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.636562 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f56affaf-b38d-4ea4-ac8c-309edaa8852e-service-ca-bundle\") pod \"router-default-5669bd84dd-j7hs7\" (UID: \"f56affaf-b38d-4ea4-ac8c-309edaa8852e\") " pod="openshift-ingress/router-default-5669bd84dd-j7hs7" Jul 19 07:42:50.636856 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.636593 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9d4e8a76-432d-4750-bb41-9d86bdb458b9-registry-certificates\") pod \"image-registry-7dbdbbf55d-6mrcs\" (UID: \"9d4e8a76-432d-4750-bb41-9d86bdb458b9\") " pod="openshift-image-registry/image-registry-7dbdbbf55d-6mrcs" Jul 19 07:42:50.636856 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:50.636706 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f56affaf-b38d-4ea4-ac8c-309edaa8852e-service-ca-bundle podName:f56affaf-b38d-4ea4-ac8c-309edaa8852e nodeName:}" failed. No retries permitted until 2026-07-19 07:42:51.13668998 +0000 UTC m=+37.490999580 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f56affaf-b38d-4ea4-ac8c-309edaa8852e-service-ca-bundle") pod "router-default-5669bd84dd-j7hs7" (UID: "f56affaf-b38d-4ea4-ac8c-309edaa8852e") : configmap references non-existent config key: service-ca.crt Jul 19 07:42:50.636856 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.636722 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fc2e627b-03e6-48c5-a476-00eaa505368a-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-2hx2z\" (UID: \"fc2e627b-03e6-48c5-a476-00eaa505368a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-2hx2z" Jul 19 07:42:50.636856 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.636733 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0adaffaf-8ab7-4966-abd1-5bbd8a0ba882-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-xr7x7\" (UID: \"0adaffaf-8ab7-4966-abd1-5bbd8a0ba882\") " pod="openshift-insights/insights-operator-555d9d7885-xr7x7" Jul 19 07:42:50.636856 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.636761 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/0adaffaf-8ab7-4966-abd1-5bbd8a0ba882-snapshots\") pod \"insights-operator-555d9d7885-xr7x7\" (UID: \"0adaffaf-8ab7-4966-abd1-5bbd8a0ba882\") " pod="openshift-insights/insights-operator-555d9d7885-xr7x7" Jul 19 07:42:50.636856 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.636766 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e990c17f-7361-430a-abe2-8fe27dc88fd6-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-m5q94\" (UID: \"e990c17f-7361-430a-abe2-8fe27dc88fd6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-m5q94" Jul 19 07:42:50.636856 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.636815 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9d4e8a76-432d-4750-bb41-9d86bdb458b9-bound-sa-token\") pod \"image-registry-7dbdbbf55d-6mrcs\" (UID: \"9d4e8a76-432d-4750-bb41-9d86bdb458b9\") " pod="openshift-image-registry/image-registry-7dbdbbf55d-6mrcs" Jul 19 07:42:50.636856 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.636841 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dd68d31d-f763-433a-867b-65b12eabac76-config\") pod \"console-operator-565b7b569d-wmhfd\" (UID: \"dd68d31d-f763-433a-867b-65b12eabac76\") " pod="openshift-console-operator/console-operator-565b7b569d-wmhfd" Jul 19 07:42:50.637311 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.636866 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7789b\" (UniqueName: \"kubernetes.io/projected/e990c17f-7361-430a-abe2-8fe27dc88fd6-kube-api-access-7789b\") pod \"cluster-samples-operator-79b7c55f87-m5q94\" (UID: \"e990c17f-7361-430a-abe2-8fe27dc88fd6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-m5q94" Jul 19 07:42:50.637311 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.636896 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/f56affaf-b38d-4ea4-ac8c-309edaa8852e-stats-auth\") pod \"router-default-5669bd84dd-j7hs7\" (UID: \"f56affaf-b38d-4ea4-ac8c-309edaa8852e\") " pod="openshift-ingress/router-default-5669bd84dd-j7hs7" Jul 19 07:42:50.637311 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.636922 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7h46q\" (UniqueName: \"kubernetes.io/projected/0adaffaf-8ab7-4966-abd1-5bbd8a0ba882-kube-api-access-7h46q\") pod \"insights-operator-555d9d7885-xr7x7\" (UID: \"0adaffaf-8ab7-4966-abd1-5bbd8a0ba882\") " pod="openshift-insights/insights-operator-555d9d7885-xr7x7" Jul 19 07:42:50.637311 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:50.636932 2574 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 19 07:42:50.637311 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.636947 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a4349d67-121d-44bb-9d73-711ee0465237-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-8w7nz\" (UID: \"a4349d67-121d-44bb-9d73-711ee0465237\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-8w7nz" Jul 19 07:42:50.637311 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.636979 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xzcd\" (UniqueName: \"kubernetes.io/projected/dd68d31d-f763-433a-867b-65b12eabac76-kube-api-access-4xzcd\") pod \"console-operator-565b7b569d-wmhfd\" (UID: \"dd68d31d-f763-433a-867b-65b12eabac76\") " pod="openshift-console-operator/console-operator-565b7b569d-wmhfd" Jul 19 07:42:50.637311 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:50.636995 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e990c17f-7361-430a-abe2-8fe27dc88fd6-samples-operator-tls podName:e990c17f-7361-430a-abe2-8fe27dc88fd6 nodeName:}" failed. No retries permitted until 2026-07-19 07:42:51.136979655 +0000 UTC m=+37.491289257 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e990c17f-7361-430a-abe2-8fe27dc88fd6-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-m5q94" (UID: "e990c17f-7361-430a-abe2-8fe27dc88fd6") : secret "samples-operator-tls" not found Jul 19 07:42:50.637311 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.637028 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9d4e8a76-432d-4750-bb41-9d86bdb458b9-ca-trust-extracted\") pod \"image-registry-7dbdbbf55d-6mrcs\" (UID: \"9d4e8a76-432d-4750-bb41-9d86bdb458b9\") " pod="openshift-image-registry/image-registry-7dbdbbf55d-6mrcs" Jul 19 07:42:50.637311 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.637063 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0adaffaf-8ab7-4966-abd1-5bbd8a0ba882-tmp\") pod \"insights-operator-555d9d7885-xr7x7\" (UID: \"0adaffaf-8ab7-4966-abd1-5bbd8a0ba882\") " pod="openshift-insights/insights-operator-555d9d7885-xr7x7" Jul 19 07:42:50.637311 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.637084 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9d4e8a76-432d-4750-bb41-9d86bdb458b9-registry-certificates\") pod \"image-registry-7dbdbbf55d-6mrcs\" (UID: \"9d4e8a76-432d-4750-bb41-9d86bdb458b9\") " pod="openshift-image-registry/image-registry-7dbdbbf55d-6mrcs" Jul 19 07:42:50.637311 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.637105 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0adaffaf-8ab7-4966-abd1-5bbd8a0ba882-service-ca-bundle\") pod \"insights-operator-555d9d7885-xr7x7\" (UID: \"0adaffaf-8ab7-4966-abd1-5bbd8a0ba882\") " pod="openshift-insights/insights-operator-555d9d7885-xr7x7" Jul 19 07:42:50.637311 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.637134 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/01c43e18-77be-4ec4-88b3-6a5662c66917-serving-cert\") pod \"service-ca-operator-5bcbc94c78-hsd5x\" (UID: \"01c43e18-77be-4ec4-88b3-6a5662c66917\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-hsd5x" Jul 19 07:42:50.637311 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.637196 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fc2e627b-03e6-48c5-a476-00eaa505368a-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-2hx2z\" (UID: \"fc2e627b-03e6-48c5-a476-00eaa505368a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-2hx2z" Jul 19 07:42:50.637311 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.637237 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9d4e8a76-432d-4750-bb41-9d86bdb458b9-registry-tls\") pod \"image-registry-7dbdbbf55d-6mrcs\" (UID: \"9d4e8a76-432d-4750-bb41-9d86bdb458b9\") " pod="openshift-image-registry/image-registry-7dbdbbf55d-6mrcs" Jul 19 07:42:50.637311 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.637239 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9d4e8a76-432d-4750-bb41-9d86bdb458b9-trusted-ca\") pod \"image-registry-7dbdbbf55d-6mrcs\" (UID: \"9d4e8a76-432d-4750-bb41-9d86bdb458b9\") " pod="openshift-image-registry/image-registry-7dbdbbf55d-6mrcs" Jul 19 07:42:50.637311 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.637265 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4821058e-2da0-4086-8911-6b394fb44223-metrics-tls\") pod \"dns-default-l7s85\" (UID: \"4821058e-2da0-4086-8911-6b394fb44223\") " pod="openshift-dns/dns-default-l7s85" Jul 19 07:42:50.638001 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.637301 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4821058e-2da0-4086-8911-6b394fb44223-tmp-dir\") pod \"dns-default-l7s85\" (UID: \"4821058e-2da0-4086-8911-6b394fb44223\") " pod="openshift-dns/dns-default-l7s85" Jul 19 07:42:50.638001 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.637326 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxsm4\" (UniqueName: \"kubernetes.io/projected/01c43e18-77be-4ec4-88b3-6a5662c66917-kube-api-access-cxsm4\") pod \"service-ca-operator-5bcbc94c78-hsd5x\" (UID: \"01c43e18-77be-4ec4-88b3-6a5662c66917\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-hsd5x" Jul 19 07:42:50.638001 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.637365 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5tmt\" (UniqueName: \"kubernetes.io/projected/33646489-0313-4d23-96c3-4b32f50233af-kube-api-access-v5tmt\") pod \"ingress-canary-m78pm\" (UID: \"33646489-0313-4d23-96c3-4b32f50233af\") " pod="openshift-ingress-canary/ingress-canary-m78pm" Jul 19 07:42:50.638001 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.637389 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ls5bd\" (UniqueName: \"kubernetes.io/projected/76040636-5309-4d6c-a14c-ffdc89085c2d-kube-api-access-ls5bd\") pod \"network-check-source-7bc5b6bf85-x2qps\" (UID: \"76040636-5309-4d6c-a14c-ffdc89085c2d\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-x2qps" Jul 19 07:42:50.638001 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.637418 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9d4e8a76-432d-4750-bb41-9d86bdb458b9-image-registry-private-configuration\") pod \"image-registry-7dbdbbf55d-6mrcs\" (UID: \"9d4e8a76-432d-4750-bb41-9d86bdb458b9\") " pod="openshift-image-registry/image-registry-7dbdbbf55d-6mrcs" Jul 19 07:42:50.638001 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.637434 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/fbd767a8-d8bc-4f92-a449-46474f1ac52d-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-5sgkp\" (UID: \"fbd767a8-d8bc-4f92-a449-46474f1ac52d\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-5sgkp" Jul 19 07:42:50.638001 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.637446 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4821058e-2da0-4086-8911-6b394fb44223-config-volume\") pod \"dns-default-l7s85\" (UID: \"4821058e-2da0-4086-8911-6b394fb44223\") " pod="openshift-dns/dns-default-l7s85" Jul 19 07:42:50.638001 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.637464 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dd68d31d-f763-433a-867b-65b12eabac76-trusted-ca\") pod \"console-operator-565b7b569d-wmhfd\" (UID: \"dd68d31d-f763-433a-867b-65b12eabac76\") " pod="openshift-console-operator/console-operator-565b7b569d-wmhfd" Jul 19 07:42:50.638001 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.637503 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dd68d31d-f763-433a-867b-65b12eabac76-serving-cert\") pod \"console-operator-565b7b569d-wmhfd\" (UID: \"dd68d31d-f763-433a-867b-65b12eabac76\") " pod="openshift-console-operator/console-operator-565b7b569d-wmhfd" Jul 19 07:42:50.638001 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.637552 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9d4e8a76-432d-4750-bb41-9d86bdb458b9-installation-pull-secrets\") pod \"image-registry-7dbdbbf55d-6mrcs\" (UID: \"9d4e8a76-432d-4750-bb41-9d86bdb458b9\") " pod="openshift-image-registry/image-registry-7dbdbbf55d-6mrcs" Jul 19 07:42:50.638001 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.637584 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/33646489-0313-4d23-96c3-4b32f50233af-cert\") pod \"ingress-canary-m78pm\" (UID: \"33646489-0313-4d23-96c3-4b32f50233af\") " pod="openshift-ingress-canary/ingress-canary-m78pm" Jul 19 07:42:50.638001 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.637609 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mt7jb\" (UniqueName: \"kubernetes.io/projected/4821058e-2da0-4086-8911-6b394fb44223-kube-api-access-mt7jb\") pod \"dns-default-l7s85\" (UID: \"4821058e-2da0-4086-8911-6b394fb44223\") " pod="openshift-dns/dns-default-l7s85" Jul 19 07:42:50.638001 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.637617 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9d4e8a76-432d-4750-bb41-9d86bdb458b9-ca-trust-extracted\") pod \"image-registry-7dbdbbf55d-6mrcs\" (UID: \"9d4e8a76-432d-4750-bb41-9d86bdb458b9\") " pod="openshift-image-registry/image-registry-7dbdbbf55d-6mrcs" Jul 19 07:42:50.638001 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.637637 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/01c43e18-77be-4ec4-88b3-6a5662c66917-config\") pod \"service-ca-operator-5bcbc94c78-hsd5x\" (UID: \"01c43e18-77be-4ec4-88b3-6a5662c66917\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-hsd5x" Jul 19 07:42:50.638001 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.637666 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f56affaf-b38d-4ea4-ac8c-309edaa8852e-metrics-certs\") pod \"router-default-5669bd84dd-j7hs7\" (UID: \"f56affaf-b38d-4ea4-ac8c-309edaa8852e\") " pod="openshift-ingress/router-default-5669bd84dd-j7hs7" Jul 19 07:42:50.638001 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.637696 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8sjx\" (UniqueName: \"kubernetes.io/projected/fc2e627b-03e6-48c5-a476-00eaa505368a-kube-api-access-v8sjx\") pod \"kube-storage-version-migrator-operator-7658d9b749-2hx2z\" (UID: \"fc2e627b-03e6-48c5-a476-00eaa505368a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-2hx2z" Jul 19 07:42:50.638631 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.637724 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jk2q\" (UniqueName: \"kubernetes.io/projected/fbd767a8-d8bc-4f92-a449-46474f1ac52d-kube-api-access-2jk2q\") pod \"cluster-monitoring-operator-76cf8966f8-5sgkp\" (UID: \"fbd767a8-d8bc-4f92-a449-46474f1ac52d\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-5sgkp" Jul 19 07:42:50.638631 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.637979 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a4349d67-121d-44bb-9d73-711ee0465237-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-8w7nz\" (UID: \"a4349d67-121d-44bb-9d73-711ee0465237\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-8w7nz" Jul 19 07:42:50.638631 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.638052 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0adaffaf-8ab7-4966-abd1-5bbd8a0ba882-service-ca-bundle\") pod \"insights-operator-555d9d7885-xr7x7\" (UID: \"0adaffaf-8ab7-4966-abd1-5bbd8a0ba882\") " pod="openshift-insights/insights-operator-555d9d7885-xr7x7" Jul 19 07:42:50.638631 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.638455 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/01c43e18-77be-4ec4-88b3-6a5662c66917-config\") pod \"service-ca-operator-5bcbc94c78-hsd5x\" (UID: \"01c43e18-77be-4ec4-88b3-6a5662c66917\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-hsd5x" Jul 19 07:42:50.638631 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.638513 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0adaffaf-8ab7-4966-abd1-5bbd8a0ba882-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-xr7x7\" (UID: \"0adaffaf-8ab7-4966-abd1-5bbd8a0ba882\") " pod="openshift-insights/insights-operator-555d9d7885-xr7x7" Jul 19 07:42:50.638631 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:50.638555 2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 19 07:42:50.638631 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:50.638595 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f56affaf-b38d-4ea4-ac8c-309edaa8852e-metrics-certs podName:f56affaf-b38d-4ea4-ac8c-309edaa8852e nodeName:}" failed. No retries permitted until 2026-07-19 07:42:51.138582016 +0000 UTC m=+37.492891609 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f56affaf-b38d-4ea4-ac8c-309edaa8852e-metrics-certs") pod "router-default-5669bd84dd-j7hs7" (UID: "f56affaf-b38d-4ea4-ac8c-309edaa8852e") : secret "router-metrics-certs-default" not found Jul 19 07:42:50.638919 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.638757 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0adaffaf-8ab7-4966-abd1-5bbd8a0ba882-tmp\") pod \"insights-operator-555d9d7885-xr7x7\" (UID: \"0adaffaf-8ab7-4966-abd1-5bbd8a0ba882\") " pod="openshift-insights/insights-operator-555d9d7885-xr7x7" Jul 19 07:42:50.639400 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.639064 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4821058e-2da0-4086-8911-6b394fb44223-tmp-dir\") pod \"dns-default-l7s85\" (UID: \"4821058e-2da0-4086-8911-6b394fb44223\") " pod="openshift-dns/dns-default-l7s85" Jul 19 07:42:50.639400 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:50.639294 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 19 07:42:50.639400 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:50.639334 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7dbdbbf55d-6mrcs: secret "image-registry-tls" not found Jul 19 07:42:50.639400 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:50.639349 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 19 07:42:50.639400 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:50.639388 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4821058e-2da0-4086-8911-6b394fb44223-metrics-tls podName:4821058e-2da0-4086-8911-6b394fb44223 nodeName:}" failed. No retries permitted until 2026-07-19 07:42:51.139376045 +0000 UTC m=+37.493685669 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4821058e-2da0-4086-8911-6b394fb44223-metrics-tls") pod "dns-default-l7s85" (UID: "4821058e-2da0-4086-8911-6b394fb44223") : secret "dns-default-metrics-tls" not found Jul 19 07:42:50.639400 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:50.639404 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9d4e8a76-432d-4750-bb41-9d86bdb458b9-registry-tls podName:9d4e8a76-432d-4750-bb41-9d86bdb458b9 nodeName:}" failed. No retries permitted until 2026-07-19 07:42:51.139395492 +0000 UTC m=+37.493705072 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9d4e8a76-432d-4750-bb41-9d86bdb458b9-registry-tls") pod "image-registry-7dbdbbf55d-6mrcs" (UID: "9d4e8a76-432d-4750-bb41-9d86bdb458b9") : secret "image-registry-tls" not found Jul 19 07:42:50.639784 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.639487 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4821058e-2da0-4086-8911-6b394fb44223-config-volume\") pod \"dns-default-l7s85\" (UID: \"4821058e-2da0-4086-8911-6b394fb44223\") " pod="openshift-dns/dns-default-l7s85" Jul 19 07:42:50.642670 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.642646 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fc2e627b-03e6-48c5-a476-00eaa505368a-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-2hx2z\" (UID: \"fc2e627b-03e6-48c5-a476-00eaa505368a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-2hx2z" Jul 19 07:42:50.643776 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.643308 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/33646489-0313-4d23-96c3-4b32f50233af-cert\") pod \"ingress-canary-m78pm\" (UID: \"33646489-0313-4d23-96c3-4b32f50233af\") " pod="openshift-ingress-canary/ingress-canary-m78pm" Jul 19 07:42:50.643776 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.643321 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9d4e8a76-432d-4750-bb41-9d86bdb458b9-installation-pull-secrets\") pod \"image-registry-7dbdbbf55d-6mrcs\" (UID: \"9d4e8a76-432d-4750-bb41-9d86bdb458b9\") " pod="openshift-image-registry/image-registry-7dbdbbf55d-6mrcs" Jul 19 07:42:50.643776 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.643348 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/f56affaf-b38d-4ea4-ac8c-309edaa8852e-stats-auth\") pod \"router-default-5669bd84dd-j7hs7\" (UID: \"f56affaf-b38d-4ea4-ac8c-309edaa8852e\") " pod="openshift-ingress/router-default-5669bd84dd-j7hs7" Jul 19 07:42:50.643776 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.643603 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dd68d31d-f763-433a-867b-65b12eabac76-serving-cert\") pod \"console-operator-565b7b569d-wmhfd\" (UID: \"dd68d31d-f763-433a-867b-65b12eabac76\") " pod="openshift-console-operator/console-operator-565b7b569d-wmhfd" Jul 19 07:42:50.643776 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.643641 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0adaffaf-8ab7-4966-abd1-5bbd8a0ba882-serving-cert\") pod \"insights-operator-555d9d7885-xr7x7\" (UID: \"0adaffaf-8ab7-4966-abd1-5bbd8a0ba882\") " pod="openshift-insights/insights-operator-555d9d7885-xr7x7" Jul 19 07:42:50.644371 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.644319 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/01c43e18-77be-4ec4-88b3-6a5662c66917-serving-cert\") pod \"service-ca-operator-5bcbc94c78-hsd5x\" (UID: \"01c43e18-77be-4ec4-88b3-6a5662c66917\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-hsd5x" Jul 19 07:42:50.644811 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.644756 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9d4e8a76-432d-4750-bb41-9d86bdb458b9-image-registry-private-configuration\") pod \"image-registry-7dbdbbf55d-6mrcs\" (UID: \"9d4e8a76-432d-4750-bb41-9d86bdb458b9\") " pod="openshift-image-registry/image-registry-7dbdbbf55d-6mrcs" Jul 19 07:42:50.645244 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.645069 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/f56affaf-b38d-4ea4-ac8c-309edaa8852e-default-certificate\") pod \"router-default-5669bd84dd-j7hs7\" (UID: \"f56affaf-b38d-4ea4-ac8c-309edaa8852e\") " pod="openshift-ingress/router-default-5669bd84dd-j7hs7" Jul 19 07:42:50.648611 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.647690 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ntqn\" (UniqueName: \"kubernetes.io/projected/9d4e8a76-432d-4750-bb41-9d86bdb458b9-kube-api-access-5ntqn\") pod \"image-registry-7dbdbbf55d-6mrcs\" (UID: \"9d4e8a76-432d-4750-bb41-9d86bdb458b9\") " pod="openshift-image-registry/image-registry-7dbdbbf55d-6mrcs" Jul 19 07:42:50.648611 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.648239 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4cfq\" (UniqueName: \"kubernetes.io/projected/f56affaf-b38d-4ea4-ac8c-309edaa8852e-kube-api-access-t4cfq\") pod \"router-default-5669bd84dd-j7hs7\" (UID: \"f56affaf-b38d-4ea4-ac8c-309edaa8852e\") " pod="openshift-ingress/router-default-5669bd84dd-j7hs7" Jul 19 07:42:50.650229 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.650018 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9d4e8a76-432d-4750-bb41-9d86bdb458b9-bound-sa-token\") pod \"image-registry-7dbdbbf55d-6mrcs\" (UID: \"9d4e8a76-432d-4750-bb41-9d86bdb458b9\") " pod="openshift-image-registry/image-registry-7dbdbbf55d-6mrcs" Jul 19 07:42:50.650229 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.650142 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jk2q\" (UniqueName: \"kubernetes.io/projected/fbd767a8-d8bc-4f92-a449-46474f1ac52d-kube-api-access-2jk2q\") pod \"cluster-monitoring-operator-76cf8966f8-5sgkp\" (UID: \"fbd767a8-d8bc-4f92-a449-46474f1ac52d\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-5sgkp" Jul 19 07:42:50.650229 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.650157 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt9tz\" (UniqueName: \"kubernetes.io/projected/d562d83b-2730-4701-b48b-43b8dc8651cc-kube-api-access-nt9tz\") pod \"volume-data-source-validator-78cf4c6f6-rtvvb\" (UID: \"d562d83b-2730-4701-b48b-43b8dc8651cc\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-rtvvb" Jul 19 07:42:50.650494 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.650248 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ls5bd\" (UniqueName: \"kubernetes.io/projected/76040636-5309-4d6c-a14c-ffdc89085c2d-kube-api-access-ls5bd\") pod \"network-check-source-7bc5b6bf85-x2qps\" (UID: \"76040636-5309-4d6c-a14c-ffdc89085c2d\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-x2qps" Jul 19 07:42:50.650494 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.650390 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xzcd\" (UniqueName: \"kubernetes.io/projected/dd68d31d-f763-433a-867b-65b12eabac76-kube-api-access-4xzcd\") pod \"console-operator-565b7b569d-wmhfd\" (UID: \"dd68d31d-f763-433a-867b-65b12eabac76\") " pod="openshift-console-operator/console-operator-565b7b569d-wmhfd" Jul 19 07:42:50.650834 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.650812 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxsm4\" (UniqueName: \"kubernetes.io/projected/01c43e18-77be-4ec4-88b3-6a5662c66917-kube-api-access-cxsm4\") pod \"service-ca-operator-5bcbc94c78-hsd5x\" (UID: \"01c43e18-77be-4ec4-88b3-6a5662c66917\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-hsd5x" Jul 19 07:42:50.651266 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.651236 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7h46q\" (UniqueName: \"kubernetes.io/projected/0adaffaf-8ab7-4966-abd1-5bbd8a0ba882-kube-api-access-7h46q\") pod \"insights-operator-555d9d7885-xr7x7\" (UID: \"0adaffaf-8ab7-4966-abd1-5bbd8a0ba882\") " pod="openshift-insights/insights-operator-555d9d7885-xr7x7" Jul 19 07:42:50.652075 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.652050 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8sjx\" (UniqueName: \"kubernetes.io/projected/fc2e627b-03e6-48c5-a476-00eaa505368a-kube-api-access-v8sjx\") pod \"kube-storage-version-migrator-operator-7658d9b749-2hx2z\" (UID: \"fc2e627b-03e6-48c5-a476-00eaa505368a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-2hx2z" Jul 19 07:42:50.652790 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.652764 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mt7jb\" (UniqueName: \"kubernetes.io/projected/4821058e-2da0-4086-8911-6b394fb44223-kube-api-access-mt7jb\") pod \"dns-default-l7s85\" (UID: \"4821058e-2da0-4086-8911-6b394fb44223\") " pod="openshift-dns/dns-default-l7s85" Jul 19 07:42:50.652887 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.652836 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7789b\" (UniqueName: \"kubernetes.io/projected/e990c17f-7361-430a-abe2-8fe27dc88fd6-kube-api-access-7789b\") pod \"cluster-samples-operator-79b7c55f87-m5q94\" (UID: \"e990c17f-7361-430a-abe2-8fe27dc88fd6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-m5q94" Jul 19 07:42:50.653220 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.653196 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5tmt\" (UniqueName: \"kubernetes.io/projected/33646489-0313-4d23-96c3-4b32f50233af-kube-api-access-v5tmt\") pod \"ingress-canary-m78pm\" (UID: \"33646489-0313-4d23-96c3-4b32f50233af\") " pod="openshift-ingress-canary/ingress-canary-m78pm" Jul 19 07:42:50.696300 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.696266 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-2hx2z" Jul 19 07:42:50.714880 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.714835 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-rtvvb" Jul 19 07:42:50.738982 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.738941 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-565b7b569d-wmhfd" Jul 19 07:42:50.750050 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.749890 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-555d9d7885-xr7x7" Jul 19 07:42:50.776070 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.775833 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-hsd5x" Jul 19 07:42:50.794659 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.794629 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-m78pm" Jul 19 07:42:50.806446 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.802652 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-x2qps" Jul 19 07:42:50.935504 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.935374 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-2hx2z"] Jul 19 07:42:50.948043 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.947667 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-rtvvb"] Jul 19 07:42:50.976046 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:50.975946 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-wmhfd"] Jul 19 07:42:50.981591 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:50.981556 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd68d31d_f763_433a_867b_65b12eabac76.slice/crio-a62d496165dbd7ffac83b72d2a74b8c999899e7755377f98da2520cdd08dee8b WatchSource:0}: Error finding container a62d496165dbd7ffac83b72d2a74b8c999899e7755377f98da2520cdd08dee8b: Status 404 returned error can't find the container with id a62d496165dbd7ffac83b72d2a74b8c999899e7755377f98da2520cdd08dee8b Jul 19 07:42:51.009720 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:51.009634 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-555d9d7885-xr7x7"] Jul 19 07:42:51.015757 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:51.015722 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0adaffaf_8ab7_4966_abd1_5bbd8a0ba882.slice/crio-482eb54557e4f02e79dc8ccf450884d28f8f87c99f3d4cd91cf85e8e88bbfb97 WatchSource:0}: Error finding container 482eb54557e4f02e79dc8ccf450884d28f8f87c99f3d4cd91cf85e8e88bbfb97: Status 404 returned error can't find the container with id 482eb54557e4f02e79dc8ccf450884d28f8f87c99f3d4cd91cf85e8e88bbfb97 Jul 19 07:42:51.029754 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:51.029722 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-hsd5x"] Jul 19 07:42:51.034003 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:51.033975 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01c43e18_77be_4ec4_88b3_6a5662c66917.slice/crio-1e2b002cc8bc0bf98f32cf1d838127c8dbdcebc9915820e378535b0400d069f0 WatchSource:0}: Error finding container 1e2b002cc8bc0bf98f32cf1d838127c8dbdcebc9915820e378535b0400d069f0: Status 404 returned error can't find the container with id 1e2b002cc8bc0bf98f32cf1d838127c8dbdcebc9915820e378535b0400d069f0 Jul 19 07:42:51.040323 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:51.040295 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-m78pm"] Jul 19 07:42:51.043830 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:51.043803 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33646489_0313_4d23_96c3_4b32f50233af.slice/crio-5c43ac23e560c7892b39c42dfa572644c8c54e0be1bbe9644627e87891cece80 WatchSource:0}: Error finding container 5c43ac23e560c7892b39c42dfa572644c8c54e0be1bbe9644627e87891cece80: Status 404 returned error can't find the container with id 5c43ac23e560c7892b39c42dfa572644c8c54e0be1bbe9644627e87891cece80 Jul 19 07:42:51.056136 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:51.056104 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-x2qps"] Jul 19 07:42:51.058697 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:42:51.058671 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76040636_5309_4d6c_a14c_ffdc89085c2d.slice/crio-7f7ecf38806c8ca05bdb6009f396b29c2d2b35c82d2a041f4f904331a4c7ce17 WatchSource:0}: Error finding container 7f7ecf38806c8ca05bdb6009f396b29c2d2b35c82d2a041f4f904331a4c7ce17: Status 404 returned error can't find the container with id 7f7ecf38806c8ca05bdb6009f396b29c2d2b35c82d2a041f4f904331a4c7ce17 Jul 19 07:42:51.144154 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:51.144062 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9d4e8a76-432d-4750-bb41-9d86bdb458b9-registry-tls\") pod \"image-registry-7dbdbbf55d-6mrcs\" (UID: \"9d4e8a76-432d-4750-bb41-9d86bdb458b9\") " pod="openshift-image-registry/image-registry-7dbdbbf55d-6mrcs" Jul 19 07:42:51.144154 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:51.144105 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4821058e-2da0-4086-8911-6b394fb44223-metrics-tls\") pod \"dns-default-l7s85\" (UID: \"4821058e-2da0-4086-8911-6b394fb44223\") " pod="openshift-dns/dns-default-l7s85" Jul 19 07:42:51.144376 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:51.144224 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 19 07:42:51.144376 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:51.144236 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 19 07:42:51.144376 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:51.144260 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7dbdbbf55d-6mrcs: secret "image-registry-tls" not found Jul 19 07:42:51.144376 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:51.144262 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f56affaf-b38d-4ea4-ac8c-309edaa8852e-metrics-certs\") pod \"router-default-5669bd84dd-j7hs7\" (UID: \"f56affaf-b38d-4ea4-ac8c-309edaa8852e\") " pod="openshift-ingress/router-default-5669bd84dd-j7hs7" Jul 19 07:42:51.144376 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:51.144288 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4821058e-2da0-4086-8911-6b394fb44223-metrics-tls podName:4821058e-2da0-4086-8911-6b394fb44223 nodeName:}" failed. No retries permitted until 2026-07-19 07:42:52.144269587 +0000 UTC m=+38.498579186 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4821058e-2da0-4086-8911-6b394fb44223-metrics-tls") pod "dns-default-l7s85" (UID: "4821058e-2da0-4086-8911-6b394fb44223") : secret "dns-default-metrics-tls" not found Jul 19 07:42:51.144376 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:51.144311 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9d4e8a76-432d-4750-bb41-9d86bdb458b9-registry-tls podName:9d4e8a76-432d-4750-bb41-9d86bdb458b9 nodeName:}" failed. No retries permitted until 2026-07-19 07:42:52.144301752 +0000 UTC m=+38.498611338 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9d4e8a76-432d-4750-bb41-9d86bdb458b9-registry-tls") pod "image-registry-7dbdbbf55d-6mrcs" (UID: "9d4e8a76-432d-4750-bb41-9d86bdb458b9") : secret "image-registry-tls" not found Jul 19 07:42:51.144376 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:51.144350 2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 19 07:42:51.144376 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:51.144353 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fbd767a8-d8bc-4f92-a449-46474f1ac52d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-5sgkp\" (UID: \"fbd767a8-d8bc-4f92-a449-46474f1ac52d\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-5sgkp" Jul 19 07:42:51.144738 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:51.144394 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f56affaf-b38d-4ea4-ac8c-309edaa8852e-metrics-certs podName:f56affaf-b38d-4ea4-ac8c-309edaa8852e nodeName:}" failed. No retries permitted until 2026-07-19 07:42:52.144380005 +0000 UTC m=+38.498689599 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f56affaf-b38d-4ea4-ac8c-309edaa8852e-metrics-certs") pod "router-default-5669bd84dd-j7hs7" (UID: "f56affaf-b38d-4ea4-ac8c-309edaa8852e") : secret "router-metrics-certs-default" not found Jul 19 07:42:51.144738 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:51.144414 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a4349d67-121d-44bb-9d73-711ee0465237-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-8w7nz\" (UID: \"a4349d67-121d-44bb-9d73-711ee0465237\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-8w7nz" Jul 19 07:42:51.144738 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:51.144443 2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 19 07:42:51.144738 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:51.144475 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f56affaf-b38d-4ea4-ac8c-309edaa8852e-service-ca-bundle\") pod \"router-default-5669bd84dd-j7hs7\" (UID: \"f56affaf-b38d-4ea4-ac8c-309edaa8852e\") " pod="openshift-ingress/router-default-5669bd84dd-j7hs7" Jul 19 07:42:51.144738 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:51.144504 2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 19 07:42:51.144738 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:51.144551 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f56affaf-b38d-4ea4-ac8c-309edaa8852e-service-ca-bundle podName:f56affaf-b38d-4ea4-ac8c-309edaa8852e nodeName:}" failed. No retries permitted until 2026-07-19 07:42:52.144537722 +0000 UTC m=+38.498847315 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f56affaf-b38d-4ea4-ac8c-309edaa8852e-service-ca-bundle") pod "router-default-5669bd84dd-j7hs7" (UID: "f56affaf-b38d-4ea4-ac8c-309edaa8852e") : configmap references non-existent config key: service-ca.crt Jul 19 07:42:51.144738 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:51.144509 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e990c17f-7361-430a-abe2-8fe27dc88fd6-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-m5q94\" (UID: \"e990c17f-7361-430a-abe2-8fe27dc88fd6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-m5q94" Jul 19 07:42:51.144738 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:51.144568 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbd767a8-d8bc-4f92-a449-46474f1ac52d-cluster-monitoring-operator-tls podName:fbd767a8-d8bc-4f92-a449-46474f1ac52d nodeName:}" failed. No retries permitted until 2026-07-19 07:42:52.144559126 +0000 UTC m=+38.498868724 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fbd767a8-d8bc-4f92-a449-46474f1ac52d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-5sgkp" (UID: "fbd767a8-d8bc-4f92-a449-46474f1ac52d") : secret "cluster-monitoring-operator-tls" not found Jul 19 07:42:51.144738 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:51.144586 2574 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 19 07:42:51.144738 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:51.144588 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4349d67-121d-44bb-9d73-711ee0465237-networking-console-plugin-cert podName:a4349d67-121d-44bb-9d73-711ee0465237 nodeName:}" failed. No retries permitted until 2026-07-19 07:42:52.144578445 +0000 UTC m=+38.498888024 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a4349d67-121d-44bb-9d73-711ee0465237-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-8w7nz" (UID: "a4349d67-121d-44bb-9d73-711ee0465237") : secret "networking-console-plugin-cert" not found Jul 19 07:42:51.144738 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:51.144638 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e990c17f-7361-430a-abe2-8fe27dc88fd6-samples-operator-tls podName:e990c17f-7361-430a-abe2-8fe27dc88fd6 nodeName:}" failed. No retries permitted until 2026-07-19 07:42:52.144622796 +0000 UTC m=+38.498932385 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e990c17f-7361-430a-abe2-8fe27dc88fd6-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-m5q94" (UID: "e990c17f-7361-430a-abe2-8fe27dc88fd6") : secret "samples-operator-tls" not found Jul 19 07:42:51.910607 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:51.910567 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-hsd5x" event={"ID":"01c43e18-77be-4ec4-88b3-6a5662c66917","Type":"ContainerStarted","Data":"1e2b002cc8bc0bf98f32cf1d838127c8dbdcebc9915820e378535b0400d069f0"} Jul 19 07:42:51.912028 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:51.911994 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-x2qps" event={"ID":"76040636-5309-4d6c-a14c-ffdc89085c2d","Type":"ContainerStarted","Data":"7f7ecf38806c8ca05bdb6009f396b29c2d2b35c82d2a041f4f904331a4c7ce17"} Jul 19 07:42:51.913444 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:51.913351 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-xr7x7" event={"ID":"0adaffaf-8ab7-4966-abd1-5bbd8a0ba882","Type":"ContainerStarted","Data":"482eb54557e4f02e79dc8ccf450884d28f8f87c99f3d4cd91cf85e8e88bbfb97"} Jul 19 07:42:51.914763 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:51.914734 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-wmhfd" event={"ID":"dd68d31d-f763-433a-867b-65b12eabac76","Type":"ContainerStarted","Data":"a62d496165dbd7ffac83b72d2a74b8c999899e7755377f98da2520cdd08dee8b"} Jul 19 07:42:51.916176 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:51.916146 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-2hx2z" event={"ID":"fc2e627b-03e6-48c5-a476-00eaa505368a","Type":"ContainerStarted","Data":"19d22edf99f261e659f74c0d960305da4ebe81dd1bfca60f3ae909f59a7032fa"} Jul 19 07:42:51.917458 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:51.917420 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-m78pm" event={"ID":"33646489-0313-4d23-96c3-4b32f50233af","Type":"ContainerStarted","Data":"5c43ac23e560c7892b39c42dfa572644c8c54e0be1bbe9644627e87891cece80"} Jul 19 07:42:51.918778 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:51.918745 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-rtvvb" event={"ID":"d562d83b-2730-4701-b48b-43b8dc8651cc","Type":"ContainerStarted","Data":"b52b53947f5cdf7e35544fe1baa247fab6d45b311280580a5328f1a3d50a5aad"} Jul 19 07:42:52.153538 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:52.153479 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f56affaf-b38d-4ea4-ac8c-309edaa8852e-metrics-certs\") pod \"router-default-5669bd84dd-j7hs7\" (UID: \"f56affaf-b38d-4ea4-ac8c-309edaa8852e\") " pod="openshift-ingress/router-default-5669bd84dd-j7hs7" Jul 19 07:42:52.153720 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:52.153555 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fbd767a8-d8bc-4f92-a449-46474f1ac52d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-5sgkp\" (UID: \"fbd767a8-d8bc-4f92-a449-46474f1ac52d\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-5sgkp" Jul 19 07:42:52.153720 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:52.153603 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a4349d67-121d-44bb-9d73-711ee0465237-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-8w7nz\" (UID: \"a4349d67-121d-44bb-9d73-711ee0465237\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-8w7nz" Jul 19 07:42:52.153720 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:52.153661 2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 19 07:42:52.153720 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:52.153663 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f56affaf-b38d-4ea4-ac8c-309edaa8852e-service-ca-bundle\") pod \"router-default-5669bd84dd-j7hs7\" (UID: \"f56affaf-b38d-4ea4-ac8c-309edaa8852e\") " pod="openshift-ingress/router-default-5669bd84dd-j7hs7" Jul 19 07:42:52.153720 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:52.153663 2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 19 07:42:52.153953 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:52.153729 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbd767a8-d8bc-4f92-a449-46474f1ac52d-cluster-monitoring-operator-tls podName:fbd767a8-d8bc-4f92-a449-46474f1ac52d nodeName:}" failed. No retries permitted until 2026-07-19 07:42:54.15371153 +0000 UTC m=+40.508021126 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fbd767a8-d8bc-4f92-a449-46474f1ac52d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-5sgkp" (UID: "fbd767a8-d8bc-4f92-a449-46474f1ac52d") : secret "cluster-monitoring-operator-tls" not found Jul 19 07:42:52.153953 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:52.153753 2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 19 07:42:52.153953 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:52.153768 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f56affaf-b38d-4ea4-ac8c-309edaa8852e-service-ca-bundle podName:f56affaf-b38d-4ea4-ac8c-309edaa8852e nodeName:}" failed. No retries permitted until 2026-07-19 07:42:54.153747572 +0000 UTC m=+40.508057170 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f56affaf-b38d-4ea4-ac8c-309edaa8852e-service-ca-bundle") pod "router-default-5669bd84dd-j7hs7" (UID: "f56affaf-b38d-4ea4-ac8c-309edaa8852e") : configmap references non-existent config key: service-ca.crt Jul 19 07:42:52.153953 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:52.153797 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f56affaf-b38d-4ea4-ac8c-309edaa8852e-metrics-certs podName:f56affaf-b38d-4ea4-ac8c-309edaa8852e nodeName:}" failed. No retries permitted until 2026-07-19 07:42:54.153786721 +0000 UTC m=+40.508096328 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f56affaf-b38d-4ea4-ac8c-309edaa8852e-metrics-certs") pod "router-default-5669bd84dd-j7hs7" (UID: "f56affaf-b38d-4ea4-ac8c-309edaa8852e") : secret "router-metrics-certs-default" not found Jul 19 07:42:52.153953 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:52.153826 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4349d67-121d-44bb-9d73-711ee0465237-networking-console-plugin-cert podName:a4349d67-121d-44bb-9d73-711ee0465237 nodeName:}" failed. No retries permitted until 2026-07-19 07:42:54.153810207 +0000 UTC m=+40.508119797 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a4349d67-121d-44bb-9d73-711ee0465237-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-8w7nz" (UID: "a4349d67-121d-44bb-9d73-711ee0465237") : secret "networking-console-plugin-cert" not found Jul 19 07:42:52.153953 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:52.153858 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e990c17f-7361-430a-abe2-8fe27dc88fd6-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-m5q94\" (UID: \"e990c17f-7361-430a-abe2-8fe27dc88fd6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-m5q94" Jul 19 07:42:52.154246 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:52.153966 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9d4e8a76-432d-4750-bb41-9d86bdb458b9-registry-tls\") pod \"image-registry-7dbdbbf55d-6mrcs\" (UID: \"9d4e8a76-432d-4750-bb41-9d86bdb458b9\") " pod="openshift-image-registry/image-registry-7dbdbbf55d-6mrcs" Jul 19 07:42:52.154246 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:52.154000 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4821058e-2da0-4086-8911-6b394fb44223-metrics-tls\") pod \"dns-default-l7s85\" (UID: \"4821058e-2da0-4086-8911-6b394fb44223\") " pod="openshift-dns/dns-default-l7s85" Jul 19 07:42:52.154246 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:52.154037 2574 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 19 07:42:52.154246 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:52.154058 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 19 07:42:52.154246 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:52.154074 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7dbdbbf55d-6mrcs: secret "image-registry-tls" not found Jul 19 07:42:52.154246 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:52.154080 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e990c17f-7361-430a-abe2-8fe27dc88fd6-samples-operator-tls podName:e990c17f-7361-430a-abe2-8fe27dc88fd6 nodeName:}" failed. No retries permitted until 2026-07-19 07:42:54.154068472 +0000 UTC m=+40.508378055 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e990c17f-7361-430a-abe2-8fe27dc88fd6-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-m5q94" (UID: "e990c17f-7361-430a-abe2-8fe27dc88fd6") : secret "samples-operator-tls" not found Jul 19 07:42:52.154246 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:52.154099 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 19 07:42:52.154246 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:52.154119 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9d4e8a76-432d-4750-bb41-9d86bdb458b9-registry-tls podName:9d4e8a76-432d-4750-bb41-9d86bdb458b9 nodeName:}" failed. No retries permitted until 2026-07-19 07:42:54.154104747 +0000 UTC m=+40.508414342 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9d4e8a76-432d-4750-bb41-9d86bdb458b9-registry-tls") pod "image-registry-7dbdbbf55d-6mrcs" (UID: "9d4e8a76-432d-4750-bb41-9d86bdb458b9") : secret "image-registry-tls" not found Jul 19 07:42:52.154246 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:52.154136 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4821058e-2da0-4086-8911-6b394fb44223-metrics-tls podName:4821058e-2da0-4086-8911-6b394fb44223 nodeName:}" failed. No retries permitted until 2026-07-19 07:42:54.154127046 +0000 UTC m=+40.508436631 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4821058e-2da0-4086-8911-6b394fb44223-metrics-tls") pod "dns-default-l7s85" (UID: "4821058e-2da0-4086-8911-6b394fb44223") : secret "dns-default-metrics-tls" not found Jul 19 07:42:52.196160 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:52.196062 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:42:52.303037 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:52.302998 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:42:52.927778 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:52.926462 2574 generic.go:356] "Generic (PLEG): container finished" podID="6d024bd7-3f48-4487-b740-2bb4a4e197fc" containerID="6edf13b7f0328058286dcf226dfb3584d16d77c918b97bbfbff44340baf9ac58" exitCode=0 Jul 19 07:42:52.927778 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:52.926792 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lpbgg" event={"ID":"6d024bd7-3f48-4487-b740-2bb4a4e197fc","Type":"ContainerDied","Data":"6edf13b7f0328058286dcf226dfb3584d16d77c918b97bbfbff44340baf9ac58"} Jul 19 07:42:53.889541 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:53.889469 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:42:53.938291 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:53.938227 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:42:53.941143 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:53.940875 2574 generic.go:356] "Generic (PLEG): container finished" podID="6d024bd7-3f48-4487-b740-2bb4a4e197fc" containerID="355c38e20d74634b52d03a67e164714307ca637bbe5e2f5f1fe524ecadd6fb3f" exitCode=0 Jul 19 07:42:53.941143 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:53.940965 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lpbgg" event={"ID":"6d024bd7-3f48-4487-b740-2bb4a4e197fc","Type":"ContainerDied","Data":"355c38e20d74634b52d03a67e164714307ca637bbe5e2f5f1fe524ecadd6fb3f"} Jul 19 07:42:54.173761 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:54.173670 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9d4e8a76-432d-4750-bb41-9d86bdb458b9-registry-tls\") pod \"image-registry-7dbdbbf55d-6mrcs\" (UID: \"9d4e8a76-432d-4750-bb41-9d86bdb458b9\") " pod="openshift-image-registry/image-registry-7dbdbbf55d-6mrcs" Jul 19 07:42:54.173761 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:54.173730 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4821058e-2da0-4086-8911-6b394fb44223-metrics-tls\") pod \"dns-default-l7s85\" (UID: \"4821058e-2da0-4086-8911-6b394fb44223\") " pod="openshift-dns/dns-default-l7s85" Jul 19 07:42:54.173983 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:54.173779 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f56affaf-b38d-4ea4-ac8c-309edaa8852e-metrics-certs\") pod \"router-default-5669bd84dd-j7hs7\" (UID: \"f56affaf-b38d-4ea4-ac8c-309edaa8852e\") " pod="openshift-ingress/router-default-5669bd84dd-j7hs7" Jul 19 07:42:54.173983 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:54.173818 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fbd767a8-d8bc-4f92-a449-46474f1ac52d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-5sgkp\" (UID: \"fbd767a8-d8bc-4f92-a449-46474f1ac52d\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-5sgkp" Jul 19 07:42:54.173983 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:54.173862 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a4349d67-121d-44bb-9d73-711ee0465237-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-8w7nz\" (UID: \"a4349d67-121d-44bb-9d73-711ee0465237\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-8w7nz" Jul 19 07:42:54.173983 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:54.173917 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f56affaf-b38d-4ea4-ac8c-309edaa8852e-service-ca-bundle\") pod \"router-default-5669bd84dd-j7hs7\" (UID: \"f56affaf-b38d-4ea4-ac8c-309edaa8852e\") " pod="openshift-ingress/router-default-5669bd84dd-j7hs7" Jul 19 07:42:54.173983 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:54.173945 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e990c17f-7361-430a-abe2-8fe27dc88fd6-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-m5q94\" (UID: \"e990c17f-7361-430a-abe2-8fe27dc88fd6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-m5q94" Jul 19 07:42:54.174244 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:54.174128 2574 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 19 07:42:54.174244 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:54.174201 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e990c17f-7361-430a-abe2-8fe27dc88fd6-samples-operator-tls podName:e990c17f-7361-430a-abe2-8fe27dc88fd6 nodeName:}" failed. No retries permitted until 2026-07-19 07:42:58.174183081 +0000 UTC m=+44.528492662 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e990c17f-7361-430a-abe2-8fe27dc88fd6-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-m5q94" (UID: "e990c17f-7361-430a-abe2-8fe27dc88fd6") : secret "samples-operator-tls" not found Jul 19 07:42:54.174740 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:54.174608 2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 19 07:42:54.174740 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:54.174664 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 19 07:42:54.174740 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:54.174680 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7dbdbbf55d-6mrcs: secret "image-registry-tls" not found Jul 19 07:42:54.174740 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:54.174668 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4349d67-121d-44bb-9d73-711ee0465237-networking-console-plugin-cert podName:a4349d67-121d-44bb-9d73-711ee0465237 nodeName:}" failed. No retries permitted until 2026-07-19 07:42:58.174651312 +0000 UTC m=+44.528960911 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a4349d67-121d-44bb-9d73-711ee0465237-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-8w7nz" (UID: "a4349d67-121d-44bb-9d73-711ee0465237") : secret "networking-console-plugin-cert" not found Jul 19 07:42:54.175047 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:54.174748 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9d4e8a76-432d-4750-bb41-9d86bdb458b9-registry-tls podName:9d4e8a76-432d-4750-bb41-9d86bdb458b9 nodeName:}" failed. No retries permitted until 2026-07-19 07:42:58.174720528 +0000 UTC m=+44.529030119 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9d4e8a76-432d-4750-bb41-9d86bdb458b9-registry-tls") pod "image-registry-7dbdbbf55d-6mrcs" (UID: "9d4e8a76-432d-4750-bb41-9d86bdb458b9") : secret "image-registry-tls" not found Jul 19 07:42:54.175047 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:54.174607 2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 19 07:42:54.175047 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:54.174807 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbd767a8-d8bc-4f92-a449-46474f1ac52d-cluster-monitoring-operator-tls podName:fbd767a8-d8bc-4f92-a449-46474f1ac52d nodeName:}" failed. No retries permitted until 2026-07-19 07:42:58.174794554 +0000 UTC m=+44.529104136 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fbd767a8-d8bc-4f92-a449-46474f1ac52d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-5sgkp" (UID: "fbd767a8-d8bc-4f92-a449-46474f1ac52d") : secret "cluster-monitoring-operator-tls" not found Jul 19 07:42:54.175047 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:54.174801 2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 19 07:42:54.175047 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:54.174835 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 19 07:42:54.175047 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:54.174841 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f56affaf-b38d-4ea4-ac8c-309edaa8852e-metrics-certs podName:f56affaf-b38d-4ea4-ac8c-309edaa8852e nodeName:}" failed. No retries permitted until 2026-07-19 07:42:58.17483317 +0000 UTC m=+44.529142754 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f56affaf-b38d-4ea4-ac8c-309edaa8852e-metrics-certs") pod "router-default-5669bd84dd-j7hs7" (UID: "f56affaf-b38d-4ea4-ac8c-309edaa8852e") : secret "router-metrics-certs-default" not found Jul 19 07:42:54.175047 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:54.174884 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4821058e-2da0-4086-8911-6b394fb44223-metrics-tls podName:4821058e-2da0-4086-8911-6b394fb44223 nodeName:}" failed. No retries permitted until 2026-07-19 07:42:58.174871637 +0000 UTC m=+44.529181225 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4821058e-2da0-4086-8911-6b394fb44223-metrics-tls") pod "dns-default-l7s85" (UID: "4821058e-2da0-4086-8911-6b394fb44223") : secret "dns-default-metrics-tls" not found Jul 19 07:42:54.175047 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:54.174921 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f56affaf-b38d-4ea4-ac8c-309edaa8852e-service-ca-bundle podName:f56affaf-b38d-4ea4-ac8c-309edaa8852e nodeName:}" failed. No retries permitted until 2026-07-19 07:42:58.174911976 +0000 UTC m=+44.529221560 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f56affaf-b38d-4ea4-ac8c-309edaa8852e-service-ca-bundle") pod "router-default-5669bd84dd-j7hs7" (UID: "f56affaf-b38d-4ea4-ac8c-309edaa8852e") : configmap references non-existent config key: service-ca.crt Jul 19 07:42:54.949158 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:54.949092 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lpbgg" event={"ID":"6d024bd7-3f48-4487-b740-2bb4a4e197fc","Type":"ContainerStarted","Data":"337fb35c3fd0a82f26af0a6a0786d5ea5e6834d553e726af33f686d439116825"} Jul 19 07:42:54.972306 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:54.972248 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-lpbgg" podStartSLOduration=9.094159183 podStartE2EDuration="40.972229199s" totalImagesPullingTime="31.878070016s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-19 07:42:14 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:42:54.970532244 +0000 UTC m=+41.324841839" watchObservedRunningTime="2026-07-19 07:42:54.972229199 +0000 UTC m=+41.326538803" Jul 19 07:42:55.199322 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:55.199234 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:42:55.247264 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:55.247225 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:42:56.039264 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:56.039229 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:42:56.091808 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:56.091775 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:42:57.074880 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:57.074829 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:42:57.175303 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:57.175258 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:42:58.217206 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:58.217163 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f56affaf-b38d-4ea4-ac8c-309edaa8852e-metrics-certs\") pod \"router-default-5669bd84dd-j7hs7\" (UID: \"f56affaf-b38d-4ea4-ac8c-309edaa8852e\") " pod="openshift-ingress/router-default-5669bd84dd-j7hs7" Jul 19 07:42:58.218010 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:58.217237 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fbd767a8-d8bc-4f92-a449-46474f1ac52d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-5sgkp\" (UID: \"fbd767a8-d8bc-4f92-a449-46474f1ac52d\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-5sgkp" Jul 19 07:42:58.218010 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:58.217286 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a4349d67-121d-44bb-9d73-711ee0465237-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-8w7nz\" (UID: \"a4349d67-121d-44bb-9d73-711ee0465237\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-8w7nz" Jul 19 07:42:58.218010 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:58.217345 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f56affaf-b38d-4ea4-ac8c-309edaa8852e-service-ca-bundle\") pod \"router-default-5669bd84dd-j7hs7\" (UID: \"f56affaf-b38d-4ea4-ac8c-309edaa8852e\") " pod="openshift-ingress/router-default-5669bd84dd-j7hs7" Jul 19 07:42:58.218010 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:58.217373 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e990c17f-7361-430a-abe2-8fe27dc88fd6-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-m5q94\" (UID: \"e990c17f-7361-430a-abe2-8fe27dc88fd6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-m5q94" Jul 19 07:42:58.218010 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:58.217346 2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 19 07:42:58.218010 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:58.217346 2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 19 07:42:58.218010 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:58.217437 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f56affaf-b38d-4ea4-ac8c-309edaa8852e-service-ca-bundle podName:f56affaf-b38d-4ea4-ac8c-309edaa8852e nodeName:}" failed. No retries permitted until 2026-07-19 07:43:06.217416918 +0000 UTC m=+52.571726518 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f56affaf-b38d-4ea4-ac8c-309edaa8852e-service-ca-bundle") pod "router-default-5669bd84dd-j7hs7" (UID: "f56affaf-b38d-4ea4-ac8c-309edaa8852e") : configmap references non-existent config key: service-ca.crt Jul 19 07:42:58.218010 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:58.217448 2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 19 07:42:58.218010 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:58.217464 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbd767a8-d8bc-4f92-a449-46474f1ac52d-cluster-monitoring-operator-tls podName:fbd767a8-d8bc-4f92-a449-46474f1ac52d nodeName:}" failed. No retries permitted until 2026-07-19 07:43:06.21745573 +0000 UTC m=+52.571765309 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fbd767a8-d8bc-4f92-a449-46474f1ac52d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-5sgkp" (UID: "fbd767a8-d8bc-4f92-a449-46474f1ac52d") : secret "cluster-monitoring-operator-tls" not found Jul 19 07:42:58.218010 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:58.217466 2574 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 19 07:42:58.218010 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:58.217476 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f56affaf-b38d-4ea4-ac8c-309edaa8852e-metrics-certs podName:f56affaf-b38d-4ea4-ac8c-309edaa8852e nodeName:}" failed. No retries permitted until 2026-07-19 07:43:06.21747048 +0000 UTC m=+52.571780058 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f56affaf-b38d-4ea4-ac8c-309edaa8852e-metrics-certs") pod "router-default-5669bd84dd-j7hs7" (UID: "f56affaf-b38d-4ea4-ac8c-309edaa8852e") : secret "router-metrics-certs-default" not found Jul 19 07:42:58.218010 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:58.217507 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4349d67-121d-44bb-9d73-711ee0465237-networking-console-plugin-cert podName:a4349d67-121d-44bb-9d73-711ee0465237 nodeName:}" failed. No retries permitted until 2026-07-19 07:43:06.217486743 +0000 UTC m=+52.571796323 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a4349d67-121d-44bb-9d73-711ee0465237-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-8w7nz" (UID: "a4349d67-121d-44bb-9d73-711ee0465237") : secret "networking-console-plugin-cert" not found Jul 19 07:42:58.218010 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:58.217544 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e990c17f-7361-430a-abe2-8fe27dc88fd6-samples-operator-tls podName:e990c17f-7361-430a-abe2-8fe27dc88fd6 nodeName:}" failed. No retries permitted until 2026-07-19 07:43:06.217533337 +0000 UTC m=+52.571842916 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e990c17f-7361-430a-abe2-8fe27dc88fd6-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-m5q94" (UID: "e990c17f-7361-430a-abe2-8fe27dc88fd6") : secret "samples-operator-tls" not found Jul 19 07:42:58.218010 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:58.217569 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9d4e8a76-432d-4750-bb41-9d86bdb458b9-registry-tls\") pod \"image-registry-7dbdbbf55d-6mrcs\" (UID: \"9d4e8a76-432d-4750-bb41-9d86bdb458b9\") " pod="openshift-image-registry/image-registry-7dbdbbf55d-6mrcs" Jul 19 07:42:58.218484 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:42:58.217603 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4821058e-2da0-4086-8911-6b394fb44223-metrics-tls\") pod \"dns-default-l7s85\" (UID: \"4821058e-2da0-4086-8911-6b394fb44223\") " pod="openshift-dns/dns-default-l7s85" Jul 19 07:42:58.218484 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:58.217700 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 19 07:42:58.218484 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:58.217719 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7dbdbbf55d-6mrcs: secret "image-registry-tls" not found Jul 19 07:42:58.218484 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:58.217747 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 19 07:42:58.218484 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:58.217764 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9d4e8a76-432d-4750-bb41-9d86bdb458b9-registry-tls podName:9d4e8a76-432d-4750-bb41-9d86bdb458b9 nodeName:}" failed. No retries permitted until 2026-07-19 07:43:06.217749908 +0000 UTC m=+52.572059498 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9d4e8a76-432d-4750-bb41-9d86bdb458b9-registry-tls") pod "image-registry-7dbdbbf55d-6mrcs" (UID: "9d4e8a76-432d-4750-bb41-9d86bdb458b9") : secret "image-registry-tls" not found Jul 19 07:42:58.218484 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:42:58.217807 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4821058e-2da0-4086-8911-6b394fb44223-metrics-tls podName:4821058e-2da0-4086-8911-6b394fb44223 nodeName:}" failed. No retries permitted until 2026-07-19 07:43:06.217770401 +0000 UTC m=+52.572079986 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4821058e-2da0-4086-8911-6b394fb44223-metrics-tls") pod "dns-default-l7s85" (UID: "4821058e-2da0-4086-8911-6b394fb44223") : secret "dns-default-metrics-tls" not found Jul 19 07:43:00.969463 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:00.969171 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-hsd5x" event={"ID":"01c43e18-77be-4ec4-88b3-6a5662c66917","Type":"ContainerStarted","Data":"68f610967d8fe88409a5abdc14d7a5b21653de1a7ba2133eb862c98f9e53ad9c"} Jul 19 07:43:00.971407 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:00.971183 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-x2qps" event={"ID":"76040636-5309-4d6c-a14c-ffdc89085c2d","Type":"ContainerStarted","Data":"635a883014d9be1d01a6c5cf3e522e6aecbede6f452d9afa51696f1cd65772c3"} Jul 19 07:43:00.972783 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:00.972752 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-xr7x7" event={"ID":"0adaffaf-8ab7-4966-abd1-5bbd8a0ba882","Type":"ContainerStarted","Data":"8b89562136e75538ed3be5c2adae6b624ff9573d9b5a4b1be785850eb9b5cb98"} Jul 19 07:43:00.974685 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:00.974663 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wmhfd_dd68d31d-f763-433a-867b-65b12eabac76/console-operator/0.log" Jul 19 07:43:00.974799 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:00.974703 2574 generic.go:356] "Generic (PLEG): container finished" podID="dd68d31d-f763-433a-867b-65b12eabac76" containerID="a6d9d7e6c10133f34ada9ffe23dc7716dd773515a486ef867963afeadb4af592" exitCode=255 Jul 19 07:43:00.974862 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:00.974787 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-wmhfd" event={"ID":"dd68d31d-f763-433a-867b-65b12eabac76","Type":"ContainerDied","Data":"a6d9d7e6c10133f34ada9ffe23dc7716dd773515a486ef867963afeadb4af592"} Jul 19 07:43:00.974994 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:00.974971 2574 scope.go:117] "RemoveContainer" containerID="a6d9d7e6c10133f34ada9ffe23dc7716dd773515a486ef867963afeadb4af592" Jul 19 07:43:00.976720 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:00.976693 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-2hx2z" event={"ID":"fc2e627b-03e6-48c5-a476-00eaa505368a","Type":"ContainerStarted","Data":"76b92993c8f7f80bab9793d99b376a2cb304324741fd02581437ec18812f675b"} Jul 19 07:43:00.978728 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:00.978702 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-m78pm" event={"ID":"33646489-0313-4d23-96c3-4b32f50233af","Type":"ContainerStarted","Data":"49c0085f291fb715f58c7eb7c7853384fbc76d70e72fdd474f525f53e350b397"} Jul 19 07:43:00.980421 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:00.980401 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-rtvvb" event={"ID":"d562d83b-2730-4701-b48b-43b8dc8651cc","Type":"ContainerStarted","Data":"bd81691f6db5884b5a543b5a32f1a9fced123120f82c09d1e4a02597dd7e2b8d"} Jul 19 07:43:00.983906 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:00.983862 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-hsd5x" podStartSLOduration=12.914106806 podStartE2EDuration="21.983845089s" totalImagesPullingTime="9.069738283s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 07:42:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:43:00.982870748 +0000 UTC m=+47.337180351" watchObservedRunningTime="2026-07-19 07:43:00.983845089 +0000 UTC m=+47.338154683" Jul 19 07:43:01.019135 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:01.019082 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-rtvvb" podStartSLOduration=12.984690916 podStartE2EDuration="22.019065227s" totalImagesPullingTime="9.034374311s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 07:42:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:43:00.99931266 +0000 UTC m=+47.353622262" watchObservedRunningTime="2026-07-19 07:43:01.019065227 +0000 UTC m=+47.373374822" Jul 19 07:43:01.019599 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:01.019565 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-x2qps" podStartSLOduration=12.966790703000001 podStartE2EDuration="22.019556544s" totalImagesPullingTime="9.052765841s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 07:42:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:43:01.018602319 +0000 UTC m=+47.372911921" watchObservedRunningTime="2026-07-19 07:43:01.019556544 +0000 UTC m=+47.373866144" Jul 19 07:43:01.038370 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:01.038316 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-555d9d7885-xr7x7" podStartSLOduration=12.952807203 podStartE2EDuration="22.03829402s" totalImagesPullingTime="9.085486817s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 07:42:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:43:01.036894501 +0000 UTC m=+47.391204118" watchObservedRunningTime="2026-07-19 07:43:01.03829402 +0000 UTC m=+47.392603626" Jul 19 07:43:01.061321 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:01.061203 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-2hx2z" podStartSLOduration=12.91604338 podStartE2EDuration="22.061180281s" totalImagesPullingTime="9.145136901s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 07:42:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:43:01.054649128 +0000 UTC m=+47.408958723" watchObservedRunningTime="2026-07-19 07:43:01.061180281 +0000 UTC m=+47.415489885" Jul 19 07:43:01.088685 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:01.088649 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:01.093974 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:01.093922 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-m78pm" podStartSLOduration=2.035721109 podStartE2EDuration="11.093900866s" totalImagesPullingTime="9.058179757s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 07:42:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:43:01.073009526 +0000 UTC m=+47.427319127" watchObservedRunningTime="2026-07-19 07:43:01.093900866 +0000 UTC m=+47.448210471" Jul 19 07:43:01.603454 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:01.603413 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-m7frd"] Jul 19 07:43:01.628820 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:01.628786 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-m7frd"] Jul 19 07:43:01.628977 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:01.628914 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-m7frd" Jul 19 07:43:01.631454 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:01.631293 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mhks2\"" Jul 19 07:43:01.631454 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:01.631293 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 19 07:43:01.632364 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:01.632284 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 19 07:43:01.632710 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:01.632685 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:01.651356 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:01.651319 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/83634718-f9cd-428d-a341-f61661f5bec3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m7frd\" (UID: \"83634718-f9cd-428d-a341-f61661f5bec3\") " pod="openshift-insights/insights-runtime-extractor-m7frd" Jul 19 07:43:01.651548 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:01.651456 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/83634718-f9cd-428d-a341-f61661f5bec3-crio-socket\") pod \"insights-runtime-extractor-m7frd\" (UID: \"83634718-f9cd-428d-a341-f61661f5bec3\") " pod="openshift-insights/insights-runtime-extractor-m7frd" Jul 19 07:43:01.651548 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:01.651490 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/83634718-f9cd-428d-a341-f61661f5bec3-data-volume\") pod \"insights-runtime-extractor-m7frd\" (UID: \"83634718-f9cd-428d-a341-f61661f5bec3\") " pod="openshift-insights/insights-runtime-extractor-m7frd" Jul 19 07:43:01.651667 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:01.651609 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/83634718-f9cd-428d-a341-f61661f5bec3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m7frd\" (UID: \"83634718-f9cd-428d-a341-f61661f5bec3\") " pod="openshift-insights/insights-runtime-extractor-m7frd" Jul 19 07:43:01.651743 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:01.651723 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6s2dm\" (UniqueName: \"kubernetes.io/projected/83634718-f9cd-428d-a341-f61661f5bec3-kube-api-access-6s2dm\") pod \"insights-runtime-extractor-m7frd\" (UID: \"83634718-f9cd-428d-a341-f61661f5bec3\") " pod="openshift-insights/insights-runtime-extractor-m7frd" Jul 19 07:43:01.702364 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:01.702327 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:01.732788 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:01.732752 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-lxg2w"] Jul 19 07:43:01.752384 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:01.752344 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/83634718-f9cd-428d-a341-f61661f5bec3-data-volume\") pod \"insights-runtime-extractor-m7frd\" (UID: \"83634718-f9cd-428d-a341-f61661f5bec3\") " pod="openshift-insights/insights-runtime-extractor-m7frd" Jul 19 07:43:01.752591 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:01.752437 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/83634718-f9cd-428d-a341-f61661f5bec3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m7frd\" (UID: \"83634718-f9cd-428d-a341-f61661f5bec3\") " pod="openshift-insights/insights-runtime-extractor-m7frd" Jul 19 07:43:01.752591 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:01.752568 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6s2dm\" (UniqueName: \"kubernetes.io/projected/83634718-f9cd-428d-a341-f61661f5bec3-kube-api-access-6s2dm\") pod \"insights-runtime-extractor-m7frd\" (UID: \"83634718-f9cd-428d-a341-f61661f5bec3\") " pod="openshift-insights/insights-runtime-extractor-m7frd" Jul 19 07:43:01.752721 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:01.752625 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-lxg2w"] Jul 19 07:43:01.752721 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:43:01.752639 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 19 07:43:01.752814 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:43:01.752720 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83634718-f9cd-428d-a341-f61661f5bec3-insights-runtime-extractor-tls podName:83634718-f9cd-428d-a341-f61661f5bec3 nodeName:}" failed. No retries permitted until 2026-07-19 07:43:02.252696567 +0000 UTC m=+48.607006170 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/83634718-f9cd-428d-a341-f61661f5bec3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-m7frd" (UID: "83634718-f9cd-428d-a341-f61661f5bec3") : secret "insights-runtime-extractor-tls" not found Jul 19 07:43:01.752814 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:01.752740 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-lxg2w" Jul 19 07:43:01.752906 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:01.752737 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/83634718-f9cd-428d-a341-f61661f5bec3-data-volume\") pod \"insights-runtime-extractor-m7frd\" (UID: \"83634718-f9cd-428d-a341-f61661f5bec3\") " pod="openshift-insights/insights-runtime-extractor-m7frd" Jul 19 07:43:01.753072 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:01.753044 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/83634718-f9cd-428d-a341-f61661f5bec3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m7frd\" (UID: \"83634718-f9cd-428d-a341-f61661f5bec3\") " pod="openshift-insights/insights-runtime-extractor-m7frd" Jul 19 07:43:01.753219 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:43:01.753110 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 19 07:43:01.753281 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:43:01.753263 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/83634718-f9cd-428d-a341-f61661f5bec3-kube-rbac-proxy-cm podName:83634718-f9cd-428d-a341-f61661f5bec3 nodeName:}" failed. No retries permitted until 2026-07-19 07:43:02.253246377 +0000 UTC m=+48.607555973 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/83634718-f9cd-428d-a341-f61661f5bec3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-m7frd" (UID: "83634718-f9cd-428d-a341-f61661f5bec3") : configmap "kube-rbac-proxy" not found Jul 19 07:43:01.753281 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:01.753198 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/83634718-f9cd-428d-a341-f61661f5bec3-crio-socket\") pod \"insights-runtime-extractor-m7frd\" (UID: \"83634718-f9cd-428d-a341-f61661f5bec3\") " pod="openshift-insights/insights-runtime-extractor-m7frd" Jul 19 07:43:01.753387 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:01.753376 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/83634718-f9cd-428d-a341-f61661f5bec3-crio-socket\") pod \"insights-runtime-extractor-m7frd\" (UID: \"83634718-f9cd-428d-a341-f61661f5bec3\") " pod="openshift-insights/insights-runtime-extractor-m7frd" Jul 19 07:43:01.755206 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:01.755181 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-6kqmb\"" Jul 19 07:43:01.755301 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:01.755184 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 19 07:43:01.755503 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:01.755479 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 19 07:43:01.765419 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:01.765395 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6s2dm\" (UniqueName: \"kubernetes.io/projected/83634718-f9cd-428d-a341-f61661f5bec3-kube-api-access-6s2dm\") pod \"insights-runtime-extractor-m7frd\" (UID: \"83634718-f9cd-428d-a341-f61661f5bec3\") " pod="openshift-insights/insights-runtime-extractor-m7frd" Jul 19 07:43:01.777790 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:01.777754 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:01.854191 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:01.854096 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kk8t8\" (UniqueName: \"kubernetes.io/projected/d4372f5b-2bef-4b86-a076-8b89ef6445c9-kube-api-access-kk8t8\") pod \"migrator-7cc9c7ffd7-lxg2w\" (UID: \"d4372f5b-2bef-4b86-a076-8b89ef6445c9\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-lxg2w" Jul 19 07:43:01.906683 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:01.906643 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:01.955101 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:01.955059 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kk8t8\" (UniqueName: \"kubernetes.io/projected/d4372f5b-2bef-4b86-a076-8b89ef6445c9-kube-api-access-kk8t8\") pod \"migrator-7cc9c7ffd7-lxg2w\" (UID: \"d4372f5b-2bef-4b86-a076-8b89ef6445c9\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-lxg2w" Jul 19 07:43:01.963013 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:01.962977 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kk8t8\" (UniqueName: \"kubernetes.io/projected/d4372f5b-2bef-4b86-a076-8b89ef6445c9-kube-api-access-kk8t8\") pod \"migrator-7cc9c7ffd7-lxg2w\" (UID: \"d4372f5b-2bef-4b86-a076-8b89ef6445c9\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-lxg2w" Jul 19 07:43:01.986139 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:01.986115 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wmhfd_dd68d31d-f763-433a-867b-65b12eabac76/console-operator/1.log" Jul 19 07:43:01.986715 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:01.986698 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wmhfd_dd68d31d-f763-433a-867b-65b12eabac76/console-operator/0.log" Jul 19 07:43:01.986816 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:01.986797 2574 generic.go:356] "Generic (PLEG): container finished" podID="dd68d31d-f763-433a-867b-65b12eabac76" containerID="871904fbd54d20070d7b33782230b0c18a3639db1a9e7a84dc5cb83a65d77255" exitCode=255 Jul 19 07:43:01.986919 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:01.986877 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-wmhfd" event={"ID":"dd68d31d-f763-433a-867b-65b12eabac76","Type":"ContainerDied","Data":"871904fbd54d20070d7b33782230b0c18a3639db1a9e7a84dc5cb83a65d77255"} Jul 19 07:43:01.987035 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:01.986949 2574 scope.go:117] "RemoveContainer" containerID="a6d9d7e6c10133f34ada9ffe23dc7716dd773515a486ef867963afeadb4af592" Jul 19 07:43:01.987150 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:01.987135 2574 scope.go:117] "RemoveContainer" containerID="871904fbd54d20070d7b33782230b0c18a3639db1a9e7a84dc5cb83a65d77255" Jul 19 07:43:01.987341 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:43:01.987319 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-wmhfd_openshift-console-operator(dd68d31d-f763-433a-867b-65b12eabac76)\"" pod="openshift-console-operator/console-operator-565b7b569d-wmhfd" podUID="dd68d31d-f763-433a-867b-65b12eabac76" Jul 19 07:43:02.062880 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:02.062850 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-lxg2w" Jul 19 07:43:02.219725 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:02.219701 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-lxg2w"] Jul 19 07:43:02.222026 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:43:02.221998 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4372f5b_2bef_4b86_a076_8b89ef6445c9.slice/crio-a0f523ab4449b500c7bed254cb2e3d0b52a1a3896f0466585de25befdc5eda1b WatchSource:0}: Error finding container a0f523ab4449b500c7bed254cb2e3d0b52a1a3896f0466585de25befdc5eda1b: Status 404 returned error can't find the container with id a0f523ab4449b500c7bed254cb2e3d0b52a1a3896f0466585de25befdc5eda1b Jul 19 07:43:02.223231 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:02.223199 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:02.258315 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:02.258289 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/83634718-f9cd-428d-a341-f61661f5bec3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m7frd\" (UID: \"83634718-f9cd-428d-a341-f61661f5bec3\") " pod="openshift-insights/insights-runtime-extractor-m7frd" Jul 19 07:43:02.258481 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:02.258417 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/83634718-f9cd-428d-a341-f61661f5bec3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m7frd\" (UID: \"83634718-f9cd-428d-a341-f61661f5bec3\") " pod="openshift-insights/insights-runtime-extractor-m7frd" Jul 19 07:43:02.258481 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:43:02.258455 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 19 07:43:02.258599 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:43:02.258490 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 19 07:43:02.258599 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:43:02.258544 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83634718-f9cd-428d-a341-f61661f5bec3-insights-runtime-extractor-tls podName:83634718-f9cd-428d-a341-f61661f5bec3 nodeName:}" failed. No retries permitted until 2026-07-19 07:43:03.258505697 +0000 UTC m=+49.612815276 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/83634718-f9cd-428d-a341-f61661f5bec3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-m7frd" (UID: "83634718-f9cd-428d-a341-f61661f5bec3") : secret "insights-runtime-extractor-tls" not found Jul 19 07:43:02.258599 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:43:02.258565 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/83634718-f9cd-428d-a341-f61661f5bec3-kube-rbac-proxy-cm podName:83634718-f9cd-428d-a341-f61661f5bec3 nodeName:}" failed. No retries permitted until 2026-07-19 07:43:03.258555167 +0000 UTC m=+49.612864747 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/83634718-f9cd-428d-a341-f61661f5bec3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-m7frd" (UID: "83634718-f9cd-428d-a341-f61661f5bec3") : configmap "kube-rbac-proxy" not found Jul 19 07:43:02.273829 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:02.273795 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:02.603811 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:02.603783 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5dh7h_cecbc684-8b2f-4c00-a7ac-2966f0d95873/dns-node-resolver/0.log" Jul 19 07:43:02.654412 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:02.654367 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:02.931890 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:02.931805 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:02.993201 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:02.993162 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wmhfd_dd68d31d-f763-433a-867b-65b12eabac76/console-operator/1.log" Jul 19 07:43:02.993723 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:02.993704 2574 scope.go:117] "RemoveContainer" containerID="871904fbd54d20070d7b33782230b0c18a3639db1a9e7a84dc5cb83a65d77255" Jul 19 07:43:02.994595 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:43:02.994562 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-wmhfd_openshift-console-operator(dd68d31d-f763-433a-867b-65b12eabac76)\"" pod="openshift-console-operator/console-operator-565b7b569d-wmhfd" podUID="dd68d31d-f763-433a-867b-65b12eabac76" Jul 19 07:43:02.999448 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:02.999418 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-lxg2w" event={"ID":"d4372f5b-2bef-4b86-a076-8b89ef6445c9","Type":"ContainerStarted","Data":"a0f523ab4449b500c7bed254cb2e3d0b52a1a3896f0466585de25befdc5eda1b"} Jul 19 07:43:03.014621 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:03.014590 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:03.269766 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:03.269703 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/83634718-f9cd-428d-a341-f61661f5bec3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m7frd\" (UID: \"83634718-f9cd-428d-a341-f61661f5bec3\") " pod="openshift-insights/insights-runtime-extractor-m7frd" Jul 19 07:43:03.269976 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:03.269801 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/83634718-f9cd-428d-a341-f61661f5bec3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m7frd\" (UID: \"83634718-f9cd-428d-a341-f61661f5bec3\") " pod="openshift-insights/insights-runtime-extractor-m7frd" Jul 19 07:43:03.269976 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:43:03.269849 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 19 07:43:03.269976 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:43:03.269937 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 19 07:43:03.269976 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:43:03.269967 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/83634718-f9cd-428d-a341-f61661f5bec3-kube-rbac-proxy-cm podName:83634718-f9cd-428d-a341-f61661f5bec3 nodeName:}" failed. No retries permitted until 2026-07-19 07:43:05.26994665 +0000 UTC m=+51.624256242 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/83634718-f9cd-428d-a341-f61661f5bec3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-m7frd" (UID: "83634718-f9cd-428d-a341-f61661f5bec3") : configmap "kube-rbac-proxy" not found Jul 19 07:43:03.270212 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:43:03.270042 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83634718-f9cd-428d-a341-f61661f5bec3-insights-runtime-extractor-tls podName:83634718-f9cd-428d-a341-f61661f5bec3 nodeName:}" failed. No retries permitted until 2026-07-19 07:43:05.2700297 +0000 UTC m=+51.624339279 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/83634718-f9cd-428d-a341-f61661f5bec3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-m7frd" (UID: "83634718-f9cd-428d-a341-f61661f5bec3") : secret "insights-runtime-extractor-tls" not found Jul 19 07:43:03.603593 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:03.603510 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ctt95_67825d1c-90c9-4e7f-87c8-575b38afa13a/node-ca/0.log" Jul 19 07:43:04.003798 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:04.003753 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-lxg2w" event={"ID":"d4372f5b-2bef-4b86-a076-8b89ef6445c9","Type":"ContainerStarted","Data":"c333d1c6fc039482c4d138f031488eb0a6af4b2af8fea09eaecc0c1b461b3af3"} Jul 19 07:43:04.404636 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:04.404603 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-m78pm_33646489-0313-4d23-96c3-4b32f50233af/serve-healthcheck-canary/0.log" Jul 19 07:43:04.530479 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:04.530447 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-58598695f7-8v9f5"] Jul 19 07:43:04.546140 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:04.546114 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-8v9f5"] Jul 19 07:43:04.546301 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:04.546222 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-8v9f5" Jul 19 07:43:04.548924 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:04.548884 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 19 07:43:04.549083 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:04.549067 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 19 07:43:04.550168 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:04.550135 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 19 07:43:04.550168 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:04.550139 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-csctg\"" Jul 19 07:43:04.550359 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:04.550139 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 19 07:43:04.556647 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:04.556612 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:04.580596 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:04.580568 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/7ffef257-51dc-4d5a-9549-f66d05d7030a-signing-key\") pod \"service-ca-58598695f7-8v9f5\" (UID: \"7ffef257-51dc-4d5a-9549-f66d05d7030a\") " pod="openshift-service-ca/service-ca-58598695f7-8v9f5" Jul 19 07:43:04.580710 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:04.580653 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/7ffef257-51dc-4d5a-9549-f66d05d7030a-signing-cabundle\") pod \"service-ca-58598695f7-8v9f5\" (UID: \"7ffef257-51dc-4d5a-9549-f66d05d7030a\") " pod="openshift-service-ca/service-ca-58598695f7-8v9f5" Jul 19 07:43:04.580710 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:04.580673 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcsj6\" (UniqueName: \"kubernetes.io/projected/7ffef257-51dc-4d5a-9549-f66d05d7030a-kube-api-access-bcsj6\") pod \"service-ca-58598695f7-8v9f5\" (UID: \"7ffef257-51dc-4d5a-9549-f66d05d7030a\") " pod="openshift-service-ca/service-ca-58598695f7-8v9f5" Jul 19 07:43:04.658532 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:04.658420 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:04.681769 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:04.681734 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/7ffef257-51dc-4d5a-9549-f66d05d7030a-signing-cabundle\") pod \"service-ca-58598695f7-8v9f5\" (UID: \"7ffef257-51dc-4d5a-9549-f66d05d7030a\") " pod="openshift-service-ca/service-ca-58598695f7-8v9f5" Jul 19 07:43:04.681928 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:04.681774 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcsj6\" (UniqueName: \"kubernetes.io/projected/7ffef257-51dc-4d5a-9549-f66d05d7030a-kube-api-access-bcsj6\") pod \"service-ca-58598695f7-8v9f5\" (UID: \"7ffef257-51dc-4d5a-9549-f66d05d7030a\") " pod="openshift-service-ca/service-ca-58598695f7-8v9f5" Jul 19 07:43:04.682037 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:04.682014 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/7ffef257-51dc-4d5a-9549-f66d05d7030a-signing-key\") pod \"service-ca-58598695f7-8v9f5\" (UID: \"7ffef257-51dc-4d5a-9549-f66d05d7030a\") " pod="openshift-service-ca/service-ca-58598695f7-8v9f5" Jul 19 07:43:04.682443 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:04.682427 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/7ffef257-51dc-4d5a-9549-f66d05d7030a-signing-cabundle\") pod \"service-ca-58598695f7-8v9f5\" (UID: \"7ffef257-51dc-4d5a-9549-f66d05d7030a\") " pod="openshift-service-ca/service-ca-58598695f7-8v9f5" Jul 19 07:43:04.690433 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:04.690409 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/7ffef257-51dc-4d5a-9549-f66d05d7030a-signing-key\") pod \"service-ca-58598695f7-8v9f5\" (UID: \"7ffef257-51dc-4d5a-9549-f66d05d7030a\") " pod="openshift-service-ca/service-ca-58598695f7-8v9f5" Jul 19 07:43:04.714774 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:04.714745 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcsj6\" (UniqueName: \"kubernetes.io/projected/7ffef257-51dc-4d5a-9549-f66d05d7030a-kube-api-access-bcsj6\") pod \"service-ca-58598695f7-8v9f5\" (UID: \"7ffef257-51dc-4d5a-9549-f66d05d7030a\") " pod="openshift-service-ca/service-ca-58598695f7-8v9f5" Jul 19 07:43:04.855871 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:04.855839 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-8v9f5" Jul 19 07:43:05.000369 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:05.000338 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-8v9f5"] Jul 19 07:43:05.003674 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:43:05.003637 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ffef257_51dc_4d5a_9549_f66d05d7030a.slice/crio-ef2b676c4487f1839ff0e5ded6e7fd346059e85b2366aa6e5734a98bb9dec00c WatchSource:0}: Error finding container ef2b676c4487f1839ff0e5ded6e7fd346059e85b2366aa6e5734a98bb9dec00c: Status 404 returned error can't find the container with id ef2b676c4487f1839ff0e5ded6e7fd346059e85b2366aa6e5734a98bb9dec00c Jul 19 07:43:05.012609 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:05.012575 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-8v9f5" event={"ID":"7ffef257-51dc-4d5a-9549-f66d05d7030a","Type":"ContainerStarted","Data":"ef2b676c4487f1839ff0e5ded6e7fd346059e85b2366aa6e5734a98bb9dec00c"} Jul 19 07:43:05.014288 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:05.014264 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-lxg2w" event={"ID":"d4372f5b-2bef-4b86-a076-8b89ef6445c9","Type":"ContainerStarted","Data":"dfbf88d05dda77baab6c59043ddf3602b2d235fced128acb7fdddf6e800dc1f7"} Jul 19 07:43:05.048604 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:05.048556 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-lxg2w" podStartSLOduration=2.380879884 podStartE2EDuration="4.048536892s" totalImagesPullingTime="1.667657008s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 07:43:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:43:05.047562217 +0000 UTC m=+51.401871819" watchObservedRunningTime="2026-07-19 07:43:05.048536892 +0000 UTC m=+51.402846487" Jul 19 07:43:05.288375 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:05.288329 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/83634718-f9cd-428d-a341-f61661f5bec3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m7frd\" (UID: \"83634718-f9cd-428d-a341-f61661f5bec3\") " pod="openshift-insights/insights-runtime-extractor-m7frd" Jul 19 07:43:05.288590 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:43:05.288490 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 19 07:43:05.288631 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:43:05.288610 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83634718-f9cd-428d-a341-f61661f5bec3-insights-runtime-extractor-tls podName:83634718-f9cd-428d-a341-f61661f5bec3 nodeName:}" failed. No retries permitted until 2026-07-19 07:43:09.288589889 +0000 UTC m=+55.642899468 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/83634718-f9cd-428d-a341-f61661f5bec3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-m7frd" (UID: "83634718-f9cd-428d-a341-f61661f5bec3") : secret "insights-runtime-extractor-tls" not found Jul 19 07:43:05.288697 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:05.288678 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/83634718-f9cd-428d-a341-f61661f5bec3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m7frd\" (UID: \"83634718-f9cd-428d-a341-f61661f5bec3\") " pod="openshift-insights/insights-runtime-extractor-m7frd" Jul 19 07:43:05.288802 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:43:05.288783 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 19 07:43:05.288861 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:43:05.288852 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/83634718-f9cd-428d-a341-f61661f5bec3-kube-rbac-proxy-cm podName:83634718-f9cd-428d-a341-f61661f5bec3 nodeName:}" failed. No retries permitted until 2026-07-19 07:43:09.288836943 +0000 UTC m=+55.643146527 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/83634718-f9cd-428d-a341-f61661f5bec3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-m7frd" (UID: "83634718-f9cd-428d-a341-f61661f5bec3") : configmap "kube-rbac-proxy" not found Jul 19 07:43:05.399530 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:05.399460 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:05.570238 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:05.570139 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:06.019603 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:06.019566 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-8v9f5" event={"ID":"7ffef257-51dc-4d5a-9549-f66d05d7030a","Type":"ContainerStarted","Data":"4afb995855b449aa702defd88d3c24b9974610327b331fd3c96d680d5a2a3795"} Jul 19 07:43:06.035726 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:06.035661 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-58598695f7-8v9f5" podStartSLOduration=2.0356436430000002 podStartE2EDuration="2.035643643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 07:43:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:43:06.034875722 +0000 UTC m=+52.389185322" watchObservedRunningTime="2026-07-19 07:43:06.035643643 +0000 UTC m=+52.389953244" Jul 19 07:43:06.298769 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:06.298670 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9d4e8a76-432d-4750-bb41-9d86bdb458b9-registry-tls\") pod \"image-registry-7dbdbbf55d-6mrcs\" (UID: \"9d4e8a76-432d-4750-bb41-9d86bdb458b9\") " pod="openshift-image-registry/image-registry-7dbdbbf55d-6mrcs" Jul 19 07:43:06.299077 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:06.299049 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4821058e-2da0-4086-8911-6b394fb44223-metrics-tls\") pod \"dns-default-l7s85\" (UID: \"4821058e-2da0-4086-8911-6b394fb44223\") " pod="openshift-dns/dns-default-l7s85" Jul 19 07:43:06.299226 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:06.299198 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f56affaf-b38d-4ea4-ac8c-309edaa8852e-metrics-certs\") pod \"router-default-5669bd84dd-j7hs7\" (UID: \"f56affaf-b38d-4ea4-ac8c-309edaa8852e\") " pod="openshift-ingress/router-default-5669bd84dd-j7hs7" Jul 19 07:43:06.299354 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:06.299245 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fbd767a8-d8bc-4f92-a449-46474f1ac52d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-5sgkp\" (UID: \"fbd767a8-d8bc-4f92-a449-46474f1ac52d\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-5sgkp" Jul 19 07:43:06.299354 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:43:06.299200 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 19 07:43:06.299354 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:43:06.299280 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7dbdbbf55d-6mrcs: secret "image-registry-tls" not found Jul 19 07:43:06.299354 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:43:06.299341 2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 19 07:43:06.299354 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:43:06.299347 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9d4e8a76-432d-4750-bb41-9d86bdb458b9-registry-tls podName:9d4e8a76-432d-4750-bb41-9d86bdb458b9 nodeName:}" failed. No retries permitted until 2026-07-19 07:43:22.299325426 +0000 UTC m=+68.653635025 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9d4e8a76-432d-4750-bb41-9d86bdb458b9-registry-tls") pod "image-registry-7dbdbbf55d-6mrcs" (UID: "9d4e8a76-432d-4750-bb41-9d86bdb458b9") : secret "image-registry-tls" not found Jul 19 07:43:06.299626 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:43:06.299350 2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 19 07:43:06.299626 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:43:06.299259 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 19 07:43:06.299626 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:43:06.299393 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbd767a8-d8bc-4f92-a449-46474f1ac52d-cluster-monitoring-operator-tls podName:fbd767a8-d8bc-4f92-a449-46474f1ac52d nodeName:}" failed. No retries permitted until 2026-07-19 07:43:22.299377012 +0000 UTC m=+68.653686614 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fbd767a8-d8bc-4f92-a449-46474f1ac52d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-5sgkp" (UID: "fbd767a8-d8bc-4f92-a449-46474f1ac52d") : secret "cluster-monitoring-operator-tls" not found Jul 19 07:43:06.299626 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:06.299418 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a4349d67-121d-44bb-9d73-711ee0465237-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-8w7nz\" (UID: \"a4349d67-121d-44bb-9d73-711ee0465237\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-8w7nz" Jul 19 07:43:06.299626 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:43:06.299447 2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 19 07:43:06.299626 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:43:06.299475 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f56affaf-b38d-4ea4-ac8c-309edaa8852e-metrics-certs podName:f56affaf-b38d-4ea4-ac8c-309edaa8852e nodeName:}" failed. No retries permitted until 2026-07-19 07:43:22.299460457 +0000 UTC m=+68.653770041 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f56affaf-b38d-4ea4-ac8c-309edaa8852e-metrics-certs") pod "router-default-5669bd84dd-j7hs7" (UID: "f56affaf-b38d-4ea4-ac8c-309edaa8852e") : secret "router-metrics-certs-default" not found Jul 19 07:43:06.299626 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:43:06.299495 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4821058e-2da0-4086-8911-6b394fb44223-metrics-tls podName:4821058e-2da0-4086-8911-6b394fb44223 nodeName:}" failed. No retries permitted until 2026-07-19 07:43:22.29948729 +0000 UTC m=+68.653796869 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4821058e-2da0-4086-8911-6b394fb44223-metrics-tls") pod "dns-default-l7s85" (UID: "4821058e-2da0-4086-8911-6b394fb44223") : secret "dns-default-metrics-tls" not found Jul 19 07:43:06.299626 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:43:06.299511 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4349d67-121d-44bb-9d73-711ee0465237-networking-console-plugin-cert podName:a4349d67-121d-44bb-9d73-711ee0465237 nodeName:}" failed. No retries permitted until 2026-07-19 07:43:22.29950424 +0000 UTC m=+68.653813830 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a4349d67-121d-44bb-9d73-711ee0465237-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-8w7nz" (UID: "a4349d67-121d-44bb-9d73-711ee0465237") : secret "networking-console-plugin-cert" not found Jul 19 07:43:06.299626 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:06.299579 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f56affaf-b38d-4ea4-ac8c-309edaa8852e-service-ca-bundle\") pod \"router-default-5669bd84dd-j7hs7\" (UID: \"f56affaf-b38d-4ea4-ac8c-309edaa8852e\") " pod="openshift-ingress/router-default-5669bd84dd-j7hs7" Jul 19 07:43:06.299626 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:06.299619 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e990c17f-7361-430a-abe2-8fe27dc88fd6-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-m5q94\" (UID: \"e990c17f-7361-430a-abe2-8fe27dc88fd6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-m5q94" Jul 19 07:43:06.300022 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:43:06.299793 2574 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 19 07:43:06.300022 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:43:06.299833 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e990c17f-7361-430a-abe2-8fe27dc88fd6-samples-operator-tls podName:e990c17f-7361-430a-abe2-8fe27dc88fd6 nodeName:}" failed. No retries permitted until 2026-07-19 07:43:22.29982229 +0000 UTC m=+68.654131873 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e990c17f-7361-430a-abe2-8fe27dc88fd6-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-m5q94" (UID: "e990c17f-7361-430a-abe2-8fe27dc88fd6") : secret "samples-operator-tls" not found Jul 19 07:43:06.300022 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:43:06.299851 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f56affaf-b38d-4ea4-ac8c-309edaa8852e-service-ca-bundle podName:f56affaf-b38d-4ea4-ac8c-309edaa8852e nodeName:}" failed. No retries permitted until 2026-07-19 07:43:22.299842754 +0000 UTC m=+68.654152334 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f56affaf-b38d-4ea4-ac8c-309edaa8852e-service-ca-bundle") pod "router-default-5669bd84dd-j7hs7" (UID: "f56affaf-b38d-4ea4-ac8c-309edaa8852e") : configmap references non-existent config key: service-ca.crt Jul 19 07:43:06.384855 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:06.384816 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:06.515475 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:06.515433 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:06.616495 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:06.616405 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:06.683559 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:06.683511 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:06.797252 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:06.797200 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:07.081068 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:07.081030 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:07.202613 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:07.202576 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:07.303010 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:07.302967 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:08.061428 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:08.061394 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:08.284293 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:08.284261 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:08.470484 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:08.470435 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:08.541704 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:08.541667 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:08.685145 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:08.685111 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:08.883860 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:08.883774 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:09.081394 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:09.081358 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:09.329079 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:09.329040 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/83634718-f9cd-428d-a341-f61661f5bec3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m7frd\" (UID: \"83634718-f9cd-428d-a341-f61661f5bec3\") " pod="openshift-insights/insights-runtime-extractor-m7frd" Jul 19 07:43:09.329484 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:09.329179 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/83634718-f9cd-428d-a341-f61661f5bec3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m7frd\" (UID: \"83634718-f9cd-428d-a341-f61661f5bec3\") " pod="openshift-insights/insights-runtime-extractor-m7frd" Jul 19 07:43:09.329484 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:43:09.329193 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 19 07:43:09.329484 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:43:09.329262 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83634718-f9cd-428d-a341-f61661f5bec3-insights-runtime-extractor-tls podName:83634718-f9cd-428d-a341-f61661f5bec3 nodeName:}" failed. No retries permitted until 2026-07-19 07:43:17.329238694 +0000 UTC m=+63.683548295 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/83634718-f9cd-428d-a341-f61661f5bec3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-m7frd" (UID: "83634718-f9cd-428d-a341-f61661f5bec3") : secret "insights-runtime-extractor-tls" not found Jul 19 07:43:09.329747 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:09.329730 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/83634718-f9cd-428d-a341-f61661f5bec3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m7frd\" (UID: \"83634718-f9cd-428d-a341-f61661f5bec3\") " pod="openshift-insights/insights-runtime-extractor-m7frd" Jul 19 07:43:10.082075 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:10.082030 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:10.182446 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:10.182409 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:10.477467 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:10.477433 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:10.685556 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:10.685499 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:10.739601 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:10.739496 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-wmhfd" Jul 19 07:43:10.739808 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:10.739608 2574 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-565b7b569d-wmhfd" Jul 19 07:43:10.740084 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:10.740070 2574 scope.go:117] "RemoveContainer" containerID="871904fbd54d20070d7b33782230b0c18a3639db1a9e7a84dc5cb83a65d77255" Jul 19 07:43:10.740281 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:43:10.740260 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-wmhfd_openshift-console-operator(dd68d31d-f763-433a-867b-65b12eabac76)\"" pod="openshift-console-operator/console-operator-565b7b569d-wmhfd" podUID="dd68d31d-f763-433a-867b-65b12eabac76" Jul 19 07:43:10.877275 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:10.877238 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:10.928852 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:10.928820 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:11.278737 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:11.278704 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:11.349215 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:11.349175 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:11.875933 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:11.875891 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:11.976264 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:11.976228 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:12.029391 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:12.029355 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:12.485578 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:12.485542 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:12.673534 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:12.673479 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:12.729622 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:12.729586 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:12.879092 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:12.878998 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:13.075697 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:13.075651 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:13.617425 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:13.617388 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:13.718228 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:13.718194 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:14.079918 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:14.079880 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:14.480674 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:14.480635 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:14.676792 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:14.676743 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:14.881700 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:14.881615 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:15.081186 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:15.081146 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:15.881573 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:15.881538 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:15.896933 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:15.896905 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nrkg5" Jul 19 07:43:16.276898 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:16.276865 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:16.478354 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:16.478320 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:16.879192 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:16.879106 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:17.077110 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:17.077067 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:17.404487 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:17.404453 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/83634718-f9cd-428d-a341-f61661f5bec3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m7frd\" (UID: \"83634718-f9cd-428d-a341-f61661f5bec3\") " pod="openshift-insights/insights-runtime-extractor-m7frd" Jul 19 07:43:17.406798 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:17.406778 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/83634718-f9cd-428d-a341-f61661f5bec3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m7frd\" (UID: \"83634718-f9cd-428d-a341-f61661f5bec3\") " pod="openshift-insights/insights-runtime-extractor-m7frd" Jul 19 07:43:17.542673 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:17.542634 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mhks2\"" Jul 19 07:43:17.550737 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:17.550710 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-m7frd" Jul 19 07:43:17.676583 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:17.676480 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-m7frd"] Jul 19 07:43:17.680743 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:43:17.680708 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83634718_f9cd_428d_a341_f61661f5bec3.slice/crio-42e5f49e36890dc7eace199521f9963a5d972b1772249eefc4df95580e743162 WatchSource:0}: Error finding container 42e5f49e36890dc7eace199521f9963a5d972b1772249eefc4df95580e743162: Status 404 returned error can't find the container with id 42e5f49e36890dc7eace199521f9963a5d972b1772249eefc4df95580e743162 Jul 19 07:43:18.060780 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:18.060743 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-m7frd" event={"ID":"83634718-f9cd-428d-a341-f61661f5bec3","Type":"ContainerStarted","Data":"42e5f49e36890dc7eace199521f9963a5d972b1772249eefc4df95580e743162"} Jul 19 07:43:20.233834 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:20.233789 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8279f51a-80ab-4d6f-98b0-32805b4d0355-metrics-certs\") pod \"network-metrics-daemon-4r4zh\" (UID: \"8279f51a-80ab-4d6f-98b0-32805b4d0355\") " pod="openshift-multus/network-metrics-daemon-4r4zh" Jul 19 07:43:20.236683 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:20.236653 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 19 07:43:20.247260 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:20.247235 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8279f51a-80ab-4d6f-98b0-32805b4d0355-metrics-certs\") pod \"network-metrics-daemon-4r4zh\" (UID: \"8279f51a-80ab-4d6f-98b0-32805b4d0355\") " pod="openshift-multus/network-metrics-daemon-4r4zh" Jul 19 07:43:20.281686 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:20.281654 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-rrpxw\"" Jul 19 07:43:20.289607 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:20.289582 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4r4zh" Jul 19 07:43:20.335323 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:20.335274 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clglj\" (UniqueName: \"kubernetes.io/projected/8320ca86-7c1c-45ce-9fe9-ee6d779d5d4d-kube-api-access-clglj\") pod \"network-check-target-lnxd5\" (UID: \"8320ca86-7c1c-45ce-9fe9-ee6d779d5d4d\") " pod="openshift-network-diagnostics/network-check-target-lnxd5" Jul 19 07:43:20.335538 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:20.335334 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/17c9482e-aaa4-41b0-8aad-ac6f428f1f48-original-pull-secret\") pod \"global-pull-secret-syncer-p5wwk\" (UID: \"17c9482e-aaa4-41b0-8aad-ac6f428f1f48\") " pod="kube-system/global-pull-secret-syncer-p5wwk" Jul 19 07:43:20.338326 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:20.338297 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-clglj\" (UniqueName: \"kubernetes.io/projected/8320ca86-7c1c-45ce-9fe9-ee6d779d5d4d-kube-api-access-clglj\") pod \"network-check-target-lnxd5\" (UID: \"8320ca86-7c1c-45ce-9fe9-ee6d779d5d4d\") " pod="openshift-network-diagnostics/network-check-target-lnxd5" Jul 19 07:43:20.338326 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:20.338327 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 19 07:43:20.348668 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:20.348644 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/17c9482e-aaa4-41b0-8aad-ac6f428f1f48-original-pull-secret\") pod \"global-pull-secret-syncer-p5wwk\" (UID: \"17c9482e-aaa4-41b0-8aad-ac6f428f1f48\") " pod="kube-system/global-pull-secret-syncer-p5wwk" Jul 19 07:43:20.565566 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:20.565459 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p5wwk" Jul 19 07:43:20.575111 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:20.575087 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ptfjq\"" Jul 19 07:43:20.583150 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:20.583118 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lnxd5" Jul 19 07:43:22.202619 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:22.202579 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:22.258405 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:22.258369 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:22.354844 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:22.354800 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9d4e8a76-432d-4750-bb41-9d86bdb458b9-registry-tls\") pod \"image-registry-7dbdbbf55d-6mrcs\" (UID: \"9d4e8a76-432d-4750-bb41-9d86bdb458b9\") " pod="openshift-image-registry/image-registry-7dbdbbf55d-6mrcs" Jul 19 07:43:22.354844 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:22.354850 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4821058e-2da0-4086-8911-6b394fb44223-metrics-tls\") pod \"dns-default-l7s85\" (UID: \"4821058e-2da0-4086-8911-6b394fb44223\") " pod="openshift-dns/dns-default-l7s85" Jul 19 07:43:22.355051 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:22.354883 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f56affaf-b38d-4ea4-ac8c-309edaa8852e-metrics-certs\") pod \"router-default-5669bd84dd-j7hs7\" (UID: \"f56affaf-b38d-4ea4-ac8c-309edaa8852e\") " pod="openshift-ingress/router-default-5669bd84dd-j7hs7" Jul 19 07:43:22.355088 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:22.355050 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fbd767a8-d8bc-4f92-a449-46474f1ac52d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-5sgkp\" (UID: \"fbd767a8-d8bc-4f92-a449-46474f1ac52d\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-5sgkp" Jul 19 07:43:22.355122 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:22.355100 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a4349d67-121d-44bb-9d73-711ee0465237-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-8w7nz\" (UID: \"a4349d67-121d-44bb-9d73-711ee0465237\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-8w7nz" Jul 19 07:43:22.355158 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:22.355137 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f56affaf-b38d-4ea4-ac8c-309edaa8852e-service-ca-bundle\") pod \"router-default-5669bd84dd-j7hs7\" (UID: \"f56affaf-b38d-4ea4-ac8c-309edaa8852e\") " pod="openshift-ingress/router-default-5669bd84dd-j7hs7" Jul 19 07:43:22.355194 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:22.355159 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e990c17f-7361-430a-abe2-8fe27dc88fd6-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-m5q94\" (UID: \"e990c17f-7361-430a-abe2-8fe27dc88fd6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-m5q94" Jul 19 07:43:22.355945 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:22.355889 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f56affaf-b38d-4ea4-ac8c-309edaa8852e-service-ca-bundle\") pod \"router-default-5669bd84dd-j7hs7\" (UID: \"f56affaf-b38d-4ea4-ac8c-309edaa8852e\") " pod="openshift-ingress/router-default-5669bd84dd-j7hs7" Jul 19 07:43:22.357827 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:22.357795 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a4349d67-121d-44bb-9d73-711ee0465237-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-8w7nz\" (UID: \"a4349d67-121d-44bb-9d73-711ee0465237\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-8w7nz" Jul 19 07:43:22.357972 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:22.357836 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4821058e-2da0-4086-8911-6b394fb44223-metrics-tls\") pod \"dns-default-l7s85\" (UID: \"4821058e-2da0-4086-8911-6b394fb44223\") " pod="openshift-dns/dns-default-l7s85" Jul 19 07:43:22.357972 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:22.357956 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e990c17f-7361-430a-abe2-8fe27dc88fd6-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-m5q94\" (UID: \"e990c17f-7361-430a-abe2-8fe27dc88fd6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-m5q94" Jul 19 07:43:22.358089 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:22.358044 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9d4e8a76-432d-4750-bb41-9d86bdb458b9-registry-tls\") pod \"image-registry-7dbdbbf55d-6mrcs\" (UID: \"9d4e8a76-432d-4750-bb41-9d86bdb458b9\") " pod="openshift-image-registry/image-registry-7dbdbbf55d-6mrcs" Jul 19 07:43:22.358128 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:22.358107 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f56affaf-b38d-4ea4-ac8c-309edaa8852e-metrics-certs\") pod \"router-default-5669bd84dd-j7hs7\" (UID: \"f56affaf-b38d-4ea4-ac8c-309edaa8852e\") " pod="openshift-ingress/router-default-5669bd84dd-j7hs7" Jul 19 07:43:22.358705 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:22.358681 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fbd767a8-d8bc-4f92-a449-46474f1ac52d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-5sgkp\" (UID: \"fbd767a8-d8bc-4f92-a449-46474f1ac52d\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-5sgkp" Jul 19 07:43:22.520540 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:22.520488 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-lz2qf\"" Jul 19 07:43:22.527801 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:22.527768 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-m5q94" Jul 19 07:43:22.533438 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:22.533413 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-thnlm\"" Jul 19 07:43:22.541425 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:22.541403 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7dbdbbf55d-6mrcs" Jul 19 07:43:22.561032 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:22.561008 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vcblr\"" Jul 19 07:43:22.567857 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:22.567837 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-cqf77\"" Jul 19 07:43:22.568852 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:22.568834 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-l7s85" Jul 19 07:43:22.575729 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:22.575704 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5669bd84dd-j7hs7" Jul 19 07:43:22.581735 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:22.581717 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-2r5vj\"" Jul 19 07:43:22.588971 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:22.588949 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-pcq4x\"" Jul 19 07:43:22.589926 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:22.589906 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-5sgkp" Jul 19 07:43:22.597663 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:22.597645 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-8w7nz" Jul 19 07:43:22.887286 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:22.887249 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4r4zh"] Jul 19 07:43:22.893460 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:43:22.893414 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8279f51a_80ab_4d6f_98b0_32805b4d0355.slice/crio-bc79c36bed5be3271464f76bcf9d4f1a03203927a751469cea21636200e94264 WatchSource:0}: Error finding container bc79c36bed5be3271464f76bcf9d4f1a03203927a751469cea21636200e94264: Status 404 returned error can't find the container with id bc79c36bed5be3271464f76bcf9d4f1a03203927a751469cea21636200e94264 Jul 19 07:43:22.958745 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:43:22.956340 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17c9482e_aaa4_41b0_8aad_ac6f428f1f48.slice/crio-3d801fe58529bd48bb23be3c3f6b3f709d3091469634580d5af70f5ad7450dfc WatchSource:0}: Error finding container 3d801fe58529bd48bb23be3c3f6b3f709d3091469634580d5af70f5ad7450dfc: Status 404 returned error can't find the container with id 3d801fe58529bd48bb23be3c3f6b3f709d3091469634580d5af70f5ad7450dfc Jul 19 07:43:22.964502 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:22.963803 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:22.970384 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:22.970223 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-54984d48f7-tsz68"] Jul 19 07:43:23.005292 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:23.004837 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58fc97b568-m5bbt"] Jul 19 07:43:23.008117 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:23.008089 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54984d48f7-tsz68" Jul 19 07:43:23.010492 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:23.010465 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78d886b7dc-dd2w6"] Jul 19 07:43:23.011445 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:23.011224 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58fc97b568-m5bbt" Jul 19 07:43:23.012665 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:23.012614 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 19 07:43:23.012843 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:23.012713 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 19 07:43:23.012948 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:23.012863 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 19 07:43:23.012948 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:23.012899 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 19 07:43:23.015170 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:23.015150 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 19 07:43:23.015340 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:23.015319 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-jm4rq\"" Jul 19 07:43:23.016953 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:23.015849 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-p5wwk"] Jul 19 07:43:23.016953 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:23.015893 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-54984d48f7-tsz68"] Jul 19 07:43:23.016953 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:23.015916 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78d886b7dc-dd2w6"] Jul 19 07:43:23.016953 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:23.015936 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58fc97b568-m5bbt"] Jul 19 07:43:23.016953 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:23.016040 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78d886b7dc-dd2w6" Jul 19 07:43:23.018590 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:23.018571 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 19 07:43:23.018977 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:23.018959 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 19 07:43:23.019295 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:23.019280 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 19 07:43:23.019690 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:23.019674 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 19 07:43:23.054683 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:23.054657 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-lnxd5"] Jul 19 07:43:23.060046 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:23.059899 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7dbdbbf55d-6mrcs"] Jul 19 07:43:23.060554 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:23.060302 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:23.061720 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:23.061695 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-m5q94"] Jul 19 07:43:23.090387 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:23.089778 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-5sgkp"] Jul 19 07:43:23.090509 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:23.090363 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7dbdbbf55d-6mrcs" event={"ID":"9d4e8a76-432d-4750-bb41-9d86bdb458b9","Type":"ContainerStarted","Data":"4aa2156f6a5ce65ef36164de8f2d2545873055e58d211a629dd9b3c8fb7f42c8"} Jul 19 07:43:23.099072 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:23.097306 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-lnxd5" event={"ID":"8320ca86-7c1c-45ce-9fe9-ee6d779d5d4d","Type":"ContainerStarted","Data":"715d35ac1f88cf481c338a565ddb6affdf5260ef224727aea6a77826ba594f05"} Jul 19 07:43:23.101576 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:23.101532 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-m7frd" event={"ID":"83634718-f9cd-428d-a341-f61661f5bec3","Type":"ContainerStarted","Data":"f01b78227fed5c5a3220e90a791079a4d1ce222edbb42724507a34f495a0a1b9"} Jul 19 07:43:23.103544 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:23.103489 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4r4zh" event={"ID":"8279f51a-80ab-4d6f-98b0-32805b4d0355","Type":"ContainerStarted","Data":"bc79c36bed5be3271464f76bcf9d4f1a03203927a751469cea21636200e94264"} Jul 19 07:43:23.107973 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:23.107652 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-p5wwk" event={"ID":"17c9482e-aaa4-41b0-8aad-ac6f428f1f48","Type":"ContainerStarted","Data":"3d801fe58529bd48bb23be3c3f6b3f709d3091469634580d5af70f5ad7450dfc"} Jul 19 07:43:23.108897 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:23.108780 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-l7s85"] Jul 19 07:43:23.110170 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:23.110151 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5669bd84dd-j7hs7"] Jul 19 07:43:23.114275 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:43:23.114243 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4821058e_2da0_4086_8911_6b394fb44223.slice/crio-bdb28649f8838eda4d24c2454885b369833e182419cfaff2a54698d0a7ca48b7 WatchSource:0}: Error finding container bdb28649f8838eda4d24c2454885b369833e182419cfaff2a54698d0a7ca48b7: Status 404 returned error can't find the container with id bdb28649f8838eda4d24c2454885b369833e182419cfaff2a54698d0a7ca48b7 Jul 19 07:43:23.114565 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:43:23.114512 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf56affaf_b38d_4ea4_ac8c_309edaa8852e.slice/crio-2cdb567b2d545aabb4e324ec8a9772ff6f1f79b2f1f5a000c7ff6bf2135beeee WatchSource:0}: Error finding container 2cdb567b2d545aabb4e324ec8a9772ff6f1f79b2f1f5a000c7ff6bf2135beeee: Status 404 returned error can't find the container with id 2cdb567b2d545aabb4e324ec8a9772ff6f1f79b2f1f5a000c7ff6bf2135beeee Jul 19 07:43:23.136402 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:23.136268 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-8w7nz"] Jul 19 07:43:23.143103 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:23.143053 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:23.203414 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:23.203380 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/43e73bcb-20e5-411f-8e9e-3e63baacf73a-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-78d886b7dc-dd2w6\" (UID: \"43e73bcb-20e5-411f-8e9e-3e63baacf73a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78d886b7dc-dd2w6" Jul 19 07:43:23.203910 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:23.203427 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/43e73bcb-20e5-411f-8e9e-3e63baacf73a-ca\") pod \"cluster-proxy-proxy-agent-78d886b7dc-dd2w6\" (UID: \"43e73bcb-20e5-411f-8e9e-3e63baacf73a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78d886b7dc-dd2w6" Jul 19 07:43:23.203910 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:23.203475 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/78e31445-54a4-4cb3-a3b6-33bb117ecc82-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-58fc97b568-m5bbt\" (UID: \"78e31445-54a4-4cb3-a3b6-33bb117ecc82\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58fc97b568-m5bbt" Jul 19 07:43:23.203910 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:23.203499 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/43e73bcb-20e5-411f-8e9e-3e63baacf73a-hub\") pod \"cluster-proxy-proxy-agent-78d886b7dc-dd2w6\" (UID: \"43e73bcb-20e5-411f-8e9e-3e63baacf73a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78d886b7dc-dd2w6" Jul 19 07:43:23.203910 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:23.203535 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qcjc\" (UniqueName: \"kubernetes.io/projected/6628e346-c758-4767-8724-ba17ff0cca1a-kube-api-access-7qcjc\") pod \"klusterlet-addon-workmgr-54984d48f7-tsz68\" (UID: \"6628e346-c758-4767-8724-ba17ff0cca1a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54984d48f7-tsz68" Jul 19 07:43:23.203910 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:23.203562 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/6628e346-c758-4767-8724-ba17ff0cca1a-klusterlet-config\") pod \"klusterlet-addon-workmgr-54984d48f7-tsz68\" (UID: \"6628e346-c758-4767-8724-ba17ff0cca1a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54984d48f7-tsz68" Jul 19 07:43:23.203910 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:23.203645 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdng2\" (UniqueName: \"kubernetes.io/projected/43e73bcb-20e5-411f-8e9e-3e63baacf73a-kube-api-access-pdng2\") pod \"cluster-proxy-proxy-agent-78d886b7dc-dd2w6\" (UID: \"43e73bcb-20e5-411f-8e9e-3e63baacf73a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78d886b7dc-dd2w6" Jul 19 07:43:23.203910 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:23.203691 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6l6tt\" (UniqueName: \"kubernetes.io/projected/78e31445-54a4-4cb3-a3b6-33bb117ecc82-kube-api-access-6l6tt\") pod \"managed-serviceaccount-addon-agent-58fc97b568-m5bbt\" (UID: \"78e31445-54a4-4cb3-a3b6-33bb117ecc82\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58fc97b568-m5bbt" Jul 19 07:43:23.203910 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:23.203732 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6628e346-c758-4767-8724-ba17ff0cca1a-tmp\") pod \"klusterlet-addon-workmgr-54984d48f7-tsz68\" (UID: \"6628e346-c758-4767-8724-ba17ff0cca1a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54984d48f7-tsz68" Jul 19 07:43:23.203910 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:23.203768 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/43e73bcb-20e5-411f-8e9e-3e63baacf73a-additional-service-ca\") pod \"cluster-proxy-proxy-agent-78d886b7dc-dd2w6\" (UID: \"43e73bcb-20e5-411f-8e9e-3e63baacf73a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78d886b7dc-dd2w6" Jul 19 07:43:23.203910 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:23.203795 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/43e73bcb-20e5-411f-8e9e-3e63baacf73a-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-78d886b7dc-dd2w6\" (UID: \"43e73bcb-20e5-411f-8e9e-3e63baacf73a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78d886b7dc-dd2w6" Jul 19 07:43:23.208288 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:23.208247 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:23.305017 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:23.304923 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdng2\" (UniqueName: \"kubernetes.io/projected/43e73bcb-20e5-411f-8e9e-3e63baacf73a-kube-api-access-pdng2\") pod \"cluster-proxy-proxy-agent-78d886b7dc-dd2w6\" (UID: \"43e73bcb-20e5-411f-8e9e-3e63baacf73a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78d886b7dc-dd2w6" Jul 19 07:43:23.305017 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:23.304973 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6l6tt\" (UniqueName: \"kubernetes.io/projected/78e31445-54a4-4cb3-a3b6-33bb117ecc82-kube-api-access-6l6tt\") pod \"managed-serviceaccount-addon-agent-58fc97b568-m5bbt\" (UID: \"78e31445-54a4-4cb3-a3b6-33bb117ecc82\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58fc97b568-m5bbt" Jul 19 07:43:23.305017 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:23.305002 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6628e346-c758-4767-8724-ba17ff0cca1a-tmp\") pod \"klusterlet-addon-workmgr-54984d48f7-tsz68\" (UID: \"6628e346-c758-4767-8724-ba17ff0cca1a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54984d48f7-tsz68" Jul 19 07:43:23.305279 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:23.305027 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/43e73bcb-20e5-411f-8e9e-3e63baacf73a-additional-service-ca\") pod \"cluster-proxy-proxy-agent-78d886b7dc-dd2w6\" (UID: \"43e73bcb-20e5-411f-8e9e-3e63baacf73a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78d886b7dc-dd2w6" Jul 19 07:43:23.305279 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:23.305050 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/43e73bcb-20e5-411f-8e9e-3e63baacf73a-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-78d886b7dc-dd2w6\" (UID: \"43e73bcb-20e5-411f-8e9e-3e63baacf73a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78d886b7dc-dd2w6" Jul 19 07:43:23.305279 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:23.305111 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/43e73bcb-20e5-411f-8e9e-3e63baacf73a-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-78d886b7dc-dd2w6\" (UID: \"43e73bcb-20e5-411f-8e9e-3e63baacf73a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78d886b7dc-dd2w6" Jul 19 07:43:23.305279 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:23.305162 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/43e73bcb-20e5-411f-8e9e-3e63baacf73a-ca\") pod \"cluster-proxy-proxy-agent-78d886b7dc-dd2w6\" (UID: \"43e73bcb-20e5-411f-8e9e-3e63baacf73a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78d886b7dc-dd2w6" Jul 19 07:43:23.305279 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:23.305193 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/78e31445-54a4-4cb3-a3b6-33bb117ecc82-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-58fc97b568-m5bbt\" (UID: \"78e31445-54a4-4cb3-a3b6-33bb117ecc82\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58fc97b568-m5bbt" Jul 19 07:43:23.305279 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:23.305213 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/43e73bcb-20e5-411f-8e9e-3e63baacf73a-hub\") pod \"cluster-proxy-proxy-agent-78d886b7dc-dd2w6\" (UID: \"43e73bcb-20e5-411f-8e9e-3e63baacf73a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78d886b7dc-dd2w6" Jul 19 07:43:23.305279 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:23.305237 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qcjc\" (UniqueName: \"kubernetes.io/projected/6628e346-c758-4767-8724-ba17ff0cca1a-kube-api-access-7qcjc\") pod \"klusterlet-addon-workmgr-54984d48f7-tsz68\" (UID: \"6628e346-c758-4767-8724-ba17ff0cca1a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54984d48f7-tsz68" Jul 19 07:43:23.305279 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:23.305265 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/6628e346-c758-4767-8724-ba17ff0cca1a-klusterlet-config\") pod \"klusterlet-addon-workmgr-54984d48f7-tsz68\" (UID: \"6628e346-c758-4767-8724-ba17ff0cca1a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54984d48f7-tsz68" Jul 19 07:43:23.305704 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:23.305508 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6628e346-c758-4767-8724-ba17ff0cca1a-tmp\") pod \"klusterlet-addon-workmgr-54984d48f7-tsz68\" (UID: \"6628e346-c758-4767-8724-ba17ff0cca1a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54984d48f7-tsz68" Jul 19 07:43:23.305916 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:23.305863 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/43e73bcb-20e5-411f-8e9e-3e63baacf73a-additional-service-ca\") pod \"cluster-proxy-proxy-agent-78d886b7dc-dd2w6\" (UID: \"43e73bcb-20e5-411f-8e9e-3e63baacf73a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78d886b7dc-dd2w6" Jul 19 07:43:23.308408 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:23.308336 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:23.308557 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:23.308436 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/43e73bcb-20e5-411f-8e9e-3e63baacf73a-ca\") pod \"cluster-proxy-proxy-agent-78d886b7dc-dd2w6\" (UID: \"43e73bcb-20e5-411f-8e9e-3e63baacf73a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78d886b7dc-dd2w6" Jul 19 07:43:23.308557 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:23.308473 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/43e73bcb-20e5-411f-8e9e-3e63baacf73a-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-78d886b7dc-dd2w6\" (UID: \"43e73bcb-20e5-411f-8e9e-3e63baacf73a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78d886b7dc-dd2w6" Jul 19 07:43:23.308707 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:23.308678 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/43e73bcb-20e5-411f-8e9e-3e63baacf73a-hub\") pod \"cluster-proxy-proxy-agent-78d886b7dc-dd2w6\" (UID: \"43e73bcb-20e5-411f-8e9e-3e63baacf73a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78d886b7dc-dd2w6" Jul 19 07:43:23.308823 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:23.308732 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/6628e346-c758-4767-8724-ba17ff0cca1a-klusterlet-config\") pod \"klusterlet-addon-workmgr-54984d48f7-tsz68\" (UID: \"6628e346-c758-4767-8724-ba17ff0cca1a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54984d48f7-tsz68" Jul 19 07:43:23.308904 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:23.308887 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/78e31445-54a4-4cb3-a3b6-33bb117ecc82-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-58fc97b568-m5bbt\" (UID: \"78e31445-54a4-4cb3-a3b6-33bb117ecc82\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58fc97b568-m5bbt" Jul 19 07:43:23.309151 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:23.309127 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/43e73bcb-20e5-411f-8e9e-3e63baacf73a-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-78d886b7dc-dd2w6\" (UID: \"43e73bcb-20e5-411f-8e9e-3e63baacf73a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78d886b7dc-dd2w6" Jul 19 07:43:23.313170 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:23.313145 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qcjc\" (UniqueName: \"kubernetes.io/projected/6628e346-c758-4767-8724-ba17ff0cca1a-kube-api-access-7qcjc\") pod \"klusterlet-addon-workmgr-54984d48f7-tsz68\" (UID: \"6628e346-c758-4767-8724-ba17ff0cca1a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54984d48f7-tsz68" Jul 19 07:43:23.313284 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:23.313146 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdng2\" (UniqueName: \"kubernetes.io/projected/43e73bcb-20e5-411f-8e9e-3e63baacf73a-kube-api-access-pdng2\") pod \"cluster-proxy-proxy-agent-78d886b7dc-dd2w6\" (UID: \"43e73bcb-20e5-411f-8e9e-3e63baacf73a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78d886b7dc-dd2w6" Jul 19 07:43:23.313284 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:23.313157 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6l6tt\" (UniqueName: \"kubernetes.io/projected/78e31445-54a4-4cb3-a3b6-33bb117ecc82-kube-api-access-6l6tt\") pod \"managed-serviceaccount-addon-agent-58fc97b568-m5bbt\" (UID: \"78e31445-54a4-4cb3-a3b6-33bb117ecc82\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58fc97b568-m5bbt" Jul 19 07:43:23.328041 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:23.328011 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54984d48f7-tsz68" Jul 19 07:43:23.346483 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:23.346280 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58fc97b568-m5bbt" Jul 19 07:43:23.352764 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:23.352499 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78d886b7dc-dd2w6" Jul 19 07:43:23.542661 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:23.542573 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-54984d48f7-tsz68"] Jul 19 07:43:23.583925 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:23.583846 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78d886b7dc-dd2w6"] Jul 19 07:43:23.591109 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:43:23.591067 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod43e73bcb_20e5_411f_8e9e_3e63baacf73a.slice/crio-ef99ffb0edfd3967c5fafc69417e3ecfbe2ae2fe12a58516015d4636aa542892 WatchSource:0}: Error finding container ef99ffb0edfd3967c5fafc69417e3ecfbe2ae2fe12a58516015d4636aa542892: Status 404 returned error can't find the container with id ef99ffb0edfd3967c5fafc69417e3ecfbe2ae2fe12a58516015d4636aa542892 Jul 19 07:43:23.614831 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:23.614742 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58fc97b568-m5bbt"] Jul 19 07:43:23.620760 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:43:23.620051 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod78e31445_54a4_4cb3_a3b6_33bb117ecc82.slice/crio-b1cab71c39446f8cc9b18b2b55446290cc050a62e966c04ed34fadb22cfc017b WatchSource:0}: Error finding container b1cab71c39446f8cc9b18b2b55446290cc050a62e966c04ed34fadb22cfc017b: Status 404 returned error can't find the container with id b1cab71c39446f8cc9b18b2b55446290cc050a62e966c04ed34fadb22cfc017b Jul 19 07:43:24.124705 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:24.124567 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-lnxd5" event={"ID":"8320ca86-7c1c-45ce-9fe9-ee6d779d5d4d","Type":"ContainerStarted","Data":"9b00ce67b25d057add075d95349d12dae19a7da74df54947d9a61e6c1f6ed6a7"} Jul 19 07:43:24.124705 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:24.124667 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-lnxd5" Jul 19 07:43:24.128652 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:24.128564 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78d886b7dc-dd2w6" event={"ID":"43e73bcb-20e5-411f-8e9e-3e63baacf73a","Type":"ContainerStarted","Data":"ef99ffb0edfd3967c5fafc69417e3ecfbe2ae2fe12a58516015d4636aa542892"} Jul 19 07:43:24.134188 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:24.133539 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5669bd84dd-j7hs7" event={"ID":"f56affaf-b38d-4ea4-ac8c-309edaa8852e","Type":"ContainerStarted","Data":"892aa21e54ec5d04163689a094c26517f9495915479bbc63833a4887729d6659"} Jul 19 07:43:24.134188 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:24.133590 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5669bd84dd-j7hs7" event={"ID":"f56affaf-b38d-4ea4-ac8c-309edaa8852e","Type":"ContainerStarted","Data":"2cdb567b2d545aabb4e324ec8a9772ff6f1f79b2f1f5a000c7ff6bf2135beeee"} Jul 19 07:43:24.150538 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:24.147725 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-lnxd5" podStartSLOduration=70.147703365 podStartE2EDuration="1m10.147703365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 07:42:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:43:24.141861771 +0000 UTC m=+70.496171373" watchObservedRunningTime="2026-07-19 07:43:24.147703365 +0000 UTC m=+70.502012967" Jul 19 07:43:24.158539 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:24.158487 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-5sgkp" event={"ID":"fbd767a8-d8bc-4f92-a449-46474f1ac52d","Type":"ContainerStarted","Data":"e05669b004ec2eeb7b30a49d98ee07617ee73ad14a0998fcc2d9a9a216463104"} Jul 19 07:43:24.162087 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:24.162059 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58fc97b568-m5bbt" event={"ID":"78e31445-54a4-4cb3-a3b6-33bb117ecc82","Type":"ContainerStarted","Data":"b1cab71c39446f8cc9b18b2b55446290cc050a62e966c04ed34fadb22cfc017b"} Jul 19 07:43:24.171830 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:24.171737 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54984d48f7-tsz68" event={"ID":"6628e346-c758-4767-8724-ba17ff0cca1a","Type":"ContainerStarted","Data":"e5ac1fb0077c14f4a978ec3cdecdddc9f7de6e782479394d5dc815e5d2054afc"} Jul 19 07:43:24.177087 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:24.175666 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-5669bd84dd-j7hs7" podStartSLOduration=45.175647648 podStartE2EDuration="45.175647648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 07:42:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:43:24.174917066 +0000 UTC m=+70.529226670" watchObservedRunningTime="2026-07-19 07:43:24.175647648 +0000 UTC m=+70.529957249" Jul 19 07:43:24.180664 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:24.180580 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-l7s85" event={"ID":"4821058e-2da0-4086-8911-6b394fb44223","Type":"ContainerStarted","Data":"bdb28649f8838eda4d24c2454885b369833e182419cfaff2a54698d0a7ca48b7"} Jul 19 07:43:24.183232 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:24.183169 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-8w7nz" event={"ID":"a4349d67-121d-44bb-9d73-711ee0465237","Type":"ContainerStarted","Data":"3f815850dd02c7b9b89ad62096b0ce3af8baa461b27465cbf4c608369dd8a901"} Jul 19 07:43:24.185221 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:24.185158 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-m5q94" event={"ID":"e990c17f-7361-430a-abe2-8fe27dc88fd6","Type":"ContainerStarted","Data":"c3857c244194141fcb35ca1594c5e087a4cce257fd4f5cb8aaaeb86bd310ba13"} Jul 19 07:43:24.194587 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:24.193546 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7dbdbbf55d-6mrcs" event={"ID":"9d4e8a76-432d-4750-bb41-9d86bdb458b9","Type":"ContainerStarted","Data":"0e6fad5f1a5c54abce6529ac7d293ab6336e2c93bc356d855d4f7b685d4dbc3d"} Jul 19 07:43:24.194587 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:24.194511 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7dbdbbf55d-6mrcs" Jul 19 07:43:24.214017 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:24.213803 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7dbdbbf55d-6mrcs" podStartSLOduration=69.21377945 podStartE2EDuration="1m9.21377945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 07:42:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:43:24.212456229 +0000 UTC m=+70.566765834" watchObservedRunningTime="2026-07-19 07:43:24.21377945 +0000 UTC m=+70.568089054" Jul 19 07:43:24.290177 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:24.290022 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:24.576424 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:24.576350 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-5669bd84dd-j7hs7" Jul 19 07:43:24.579926 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:24.579709 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-5669bd84dd-j7hs7" Jul 19 07:43:25.203452 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:25.203421 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-5669bd84dd-j7hs7" Jul 19 07:43:25.205021 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:25.204966 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-5669bd84dd-j7hs7" Jul 19 07:43:25.553049 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:25.552619 2574 scope.go:117] "RemoveContainer" containerID="871904fbd54d20070d7b33782230b0c18a3639db1a9e7a84dc5cb83a65d77255" Jul 19 07:43:28.462838 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:28.462801 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:34.142897 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:34.142812 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:34.245352 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:34.245312 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:37.203446 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:37.203398 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:37.263107 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:37.263063 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-5sgkp" event={"ID":"fbd767a8-d8bc-4f92-a449-46474f1ac52d","Type":"ContainerStarted","Data":"fc0e1b8e16bbdb2ab8a940f199128e509ff5fceb3962f89128c592f8cab7ff4f"} Jul 19 07:43:37.265272 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:37.265240 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58fc97b568-m5bbt" event={"ID":"78e31445-54a4-4cb3-a3b6-33bb117ecc82","Type":"ContainerStarted","Data":"8d3fc7d04ed7587568e9f2fd5dad9659f1dd420e226a595397e49c339a317c14"} Jul 19 07:43:37.269123 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:37.269092 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-m7frd" event={"ID":"83634718-f9cd-428d-a341-f61661f5bec3","Type":"ContainerStarted","Data":"e659a1419adbe67b7b8b64b5aed1cb10e534a56b80831c2f5fa01ab71c913950"} Jul 19 07:43:37.271025 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:37.270994 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-p5wwk" event={"ID":"17c9482e-aaa4-41b0-8aad-ac6f428f1f48","Type":"ContainerStarted","Data":"eac2cec2eb780e49b225ef404ecda7b21beea349aedb3526c9d8c615faa2b5e8"} Jul 19 07:43:37.273101 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:37.273071 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54984d48f7-tsz68" event={"ID":"6628e346-c758-4767-8724-ba17ff0cca1a","Type":"ContainerStarted","Data":"ca20d11862d781f698b0207b5d3240b7a91dd05a6a12ab70c43d3ac841ad8066"} Jul 19 07:43:37.273295 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:37.273276 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54984d48f7-tsz68" Jul 19 07:43:37.275985 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:37.275718 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54984d48f7-tsz68" Jul 19 07:43:37.276100 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:37.276052 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-l7s85" event={"ID":"4821058e-2da0-4086-8911-6b394fb44223","Type":"ContainerStarted","Data":"bf8fe1939e94d8bb88cf77f38f87ed473f7979caff093a7393d1ae1883f931d7"} Jul 19 07:43:37.276165 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:37.276097 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-l7s85" event={"ID":"4821058e-2da0-4086-8911-6b394fb44223","Type":"ContainerStarted","Data":"5bb64fff4dd3672ea39944ae948efb0348a3e3ff0bcd9afb39549084a6ba9b67"} Jul 19 07:43:37.276165 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:37.276143 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-l7s85" Jul 19 07:43:37.278322 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:37.278295 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-8w7nz" event={"ID":"a4349d67-121d-44bb-9d73-711ee0465237","Type":"ContainerStarted","Data":"e2ef84011fa1283cad9327f24e097c378faead18677e89ee6384b43869c0d79f"} Jul 19 07:43:37.281021 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:37.280945 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-5sgkp" podStartSLOduration=46.807419969 podStartE2EDuration="58.280926047s" totalImagesPullingTime="11.473506078s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 07:42:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:43:37.279004119 +0000 UTC m=+83.633313721" watchObservedRunningTime="2026-07-19 07:43:37.280926047 +0000 UTC m=+83.635235650" Jul 19 07:43:37.281162 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:37.281024 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-m5q94" event={"ID":"e990c17f-7361-430a-abe2-8fe27dc88fd6","Type":"ContainerStarted","Data":"7bae88b06502719d56e0f329be16814c9b1565191156aaf4036e58e528c76594"} Jul 19 07:43:37.281162 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:37.281069 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-m5q94" event={"ID":"e990c17f-7361-430a-abe2-8fe27dc88fd6","Type":"ContainerStarted","Data":"a6c107bf99c0ae796b6e6b0ddf94759f2645d0c731ff5a12ad1f3531bfa3bb4f"} Jul 19 07:43:37.283827 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:37.283802 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wmhfd_dd68d31d-f763-433a-867b-65b12eabac76/console-operator/1.log" Jul 19 07:43:37.283936 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:37.283905 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-wmhfd" event={"ID":"dd68d31d-f763-433a-867b-65b12eabac76","Type":"ContainerStarted","Data":"ebcd97d5d8198cb3071325877bc55ffa7cdce2814d0dda7433956a0a8f4d0327"} Jul 19 07:43:37.284194 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:37.284153 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-wmhfd" Jul 19 07:43:37.285128 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:37.284970 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:37.286861 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:37.286832 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4r4zh" event={"ID":"8279f51a-80ab-4d6f-98b0-32805b4d0355","Type":"ContainerStarted","Data":"02159aafd170e23694046e232b91bccf2a224213e774dadaa281992c580516be"} Jul 19 07:43:37.286965 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:37.286895 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4r4zh" event={"ID":"8279f51a-80ab-4d6f-98b0-32805b4d0355","Type":"ContainerStarted","Data":"6122d917213cc80f09703636390639b702965443451767533519fd8244c87256"} Jul 19 07:43:37.289600 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:37.289564 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78d886b7dc-dd2w6" event={"ID":"43e73bcb-20e5-411f-8e9e-3e63baacf73a","Type":"ContainerStarted","Data":"7765c47438e61b241c56750b41a4d86238d9da71d7037c9fa162f65c33e0bf0b"} Jul 19 07:43:37.289706 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:37.289615 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78d886b7dc-dd2w6" event={"ID":"43e73bcb-20e5-411f-8e9e-3e63baacf73a","Type":"ContainerStarted","Data":"ef4f3439a7a9ad018408910d22b839008a875453bd77db03e9fb07533f5ee708"} Jul 19 07:43:37.289706 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:37.289632 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78d886b7dc-dd2w6" event={"ID":"43e73bcb-20e5-411f-8e9e-3e63baacf73a","Type":"ContainerStarted","Data":"c2c2b497302dc81ff75e44d1b8801c9ab293a33bb8875f19e8aaa8b478d88d30"} Jul 19 07:43:37.290843 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:37.290824 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-565b7b569d-wmhfd" Jul 19 07:43:37.296804 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:37.296757 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-l7s85" podStartSLOduration=35.155616884 podStartE2EDuration="47.296739654s" totalImagesPullingTime="12.14112277s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 07:42:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:43:37.294699002 +0000 UTC m=+83.649008606" watchObservedRunningTime="2026-07-19 07:43:37.296739654 +0000 UTC m=+83.651049256" Jul 19 07:43:37.311963 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:37.311903 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-8w7nz" podStartSLOduration=40.288387 podStartE2EDuration="51.311891451s" totalImagesPullingTime="11.023504451s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 07:42:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:43:37.311587016 +0000 UTC m=+83.665896619" watchObservedRunningTime="2026-07-19 07:43:37.311891451 +0000 UTC m=+83.666201050" Jul 19 07:43:37.339247 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:37.338708 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-p5wwk" podStartSLOduration=68.664677212 podStartE2EDuration="1m22.338687104s" totalImagesPullingTime="13.674009892s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-19 07:42:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:43:37.336643219 +0000 UTC m=+83.690952835" watchObservedRunningTime="2026-07-19 07:43:37.338687104 +0000 UTC m=+83.692996705" Jul 19 07:43:37.359540 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:37.358365 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58fc97b568-m5bbt" podStartSLOduration=2.326810186 podStartE2EDuration="15.358342855s" totalImagesPullingTime="13.031532669s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 07:43:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:43:37.357917217 +0000 UTC m=+83.712226819" watchObservedRunningTime="2026-07-19 07:43:37.358342855 +0000 UTC m=+83.712652457" Jul 19 07:43:37.383538 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:37.382497 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54984d48f7-tsz68" podStartSLOduration=2.340536438 podStartE2EDuration="15.382476734s" totalImagesPullingTime="13.041940296s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 07:43:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:43:37.380438553 +0000 UTC m=+83.734748151" watchObservedRunningTime="2026-07-19 07:43:37.382476734 +0000 UTC m=+83.736786336" Jul 19 07:43:37.408239 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:37.408185 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-4r4zh" podStartSLOduration=74.187419637 podStartE2EDuration="1m23.408164389s" totalImagesPullingTime="9.220744752s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 07:42:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:43:37.406925926 +0000 UTC m=+83.761235528" watchObservedRunningTime="2026-07-19 07:43:37.408164389 +0000 UTC m=+83.762473994" Jul 19 07:43:37.412676 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:37.412640 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:37.434242 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:37.434173 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-565b7b569d-wmhfd" podStartSLOduration=49.315420037 podStartE2EDuration="58.43415049s" totalImagesPullingTime="9.118730453s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 07:42:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:43:37.432854469 +0000 UTC m=+83.787164071" watchObservedRunningTime="2026-07-19 07:43:37.43415049 +0000 UTC m=+83.788460092" Jul 19 07:43:37.453947 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:37.453298 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78d886b7dc-dd2w6" podStartSLOduration=2.344014864 podStartE2EDuration="15.45327641s" totalImagesPullingTime="13.109261546s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 07:43:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:43:37.452490785 +0000 UTC m=+83.806800398" watchObservedRunningTime="2026-07-19 07:43:37.45327641 +0000 UTC m=+83.807586013" Jul 19 07:43:37.471424 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:37.471368 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-m5q94" podStartSLOduration=46.330289452 podStartE2EDuration="58.471343738s" totalImagesPullingTime="12.141054286s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 07:42:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:43:37.4694914 +0000 UTC m=+83.823801002" watchObservedRunningTime="2026-07-19 07:43:37.471343738 +0000 UTC m=+83.825653341" Jul 19 07:43:37.505001 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:37.504953 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:37.581230 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:37.581128 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:37.636994 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:37.636954 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:37.704505 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:37.704461 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:37.878481 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:37.878386 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:37.927542 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:37.927478 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:38.079595 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:38.079554 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:38.181343 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:38.181242 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:38.487684 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:38.487645 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:38.537354 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:38.537316 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:38.897716 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:38.897623 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:38.949907 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:38.949872 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:39.092627 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:39.092569 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:39.763302 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:39.763262 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:40.649440 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:40.649342 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:40.749661 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:40.749628 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:41.110124 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:41.110082 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:41.211074 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:41.211034 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:41.310201 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:41.310158 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-m7frd" event={"ID":"83634718-f9cd-428d-a341-f61661f5bec3","Type":"ContainerStarted","Data":"07e68e43ae2c8503fde5f27291e9111873d3bff3b9475bbb21a08f797dd3529b"} Jul 19 07:43:41.311553 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:41.311504 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:41.327593 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:41.327542 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-m7frd" podStartSLOduration=17.937673183 podStartE2EDuration="40.327509958s" totalImagesPullingTime="22.389836775s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-19 07:43:01 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:43:41.325983822 +0000 UTC m=+87.680293423" watchObservedRunningTime="2026-07-19 07:43:41.327509958 +0000 UTC m=+87.681819592" Jul 19 07:43:41.411256 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:41.411167 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:42.435686 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:42.435657 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:42.536627 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:42.536588 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:42.545726 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:42.545682 2574 patch_prober.go:28] interesting pod/image-registry-7dbdbbf55d-6mrcs container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 19 07:43:42.545890 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:42.545741 2574 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-7dbdbbf55d-6mrcs" podUID="9d4e8a76-432d-4750-bb41-9d86bdb458b9" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 19 07:43:42.669241 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:42.669198 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:45.417760 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:45.417719 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:45.830338 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:45.830293 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:45.931302 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:45.931257 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:46.092477 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:46.092385 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7dbdbbf55d-6mrcs"] Jul 19 07:43:46.096781 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:46.096745 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7dbdbbf55d-6mrcs" Jul 19 07:43:46.164127 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:46.164085 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:46.265303 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:46.265265 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:46.313391 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:46.313354 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:47.297005 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:47.296971 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-l7s85" Jul 19 07:43:47.552085 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:47.551980 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:47.768429 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:47.768389 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:47.806077 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:47.805979 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-g6cp9"] Jul 19 07:43:47.811509 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:47.811482 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-g6cp9" Jul 19 07:43:47.814371 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:47.814345 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 19 07:43:47.815444 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:47.815412 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 19 07:43:47.815444 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:47.815436 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 19 07:43:47.815661 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:47.815491 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-5pdwm\"" Jul 19 07:43:47.815661 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:47.815436 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 19 07:43:47.868534 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:47.868485 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:47.907873 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:47.907837 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgntq\" (UniqueName: \"kubernetes.io/projected/762f12f8-e846-45b9-9ba5-aafbb2aabb66-kube-api-access-dgntq\") pod \"node-exporter-g6cp9\" (UID: \"762f12f8-e846-45b9-9ba5-aafbb2aabb66\") " pod="openshift-monitoring/node-exporter-g6cp9" Jul 19 07:43:47.908087 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:47.907886 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/762f12f8-e846-45b9-9ba5-aafbb2aabb66-node-exporter-wtmp\") pod \"node-exporter-g6cp9\" (UID: \"762f12f8-e846-45b9-9ba5-aafbb2aabb66\") " pod="openshift-monitoring/node-exporter-g6cp9" Jul 19 07:43:47.908087 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:47.907911 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/762f12f8-e846-45b9-9ba5-aafbb2aabb66-metrics-client-ca\") pod \"node-exporter-g6cp9\" (UID: \"762f12f8-e846-45b9-9ba5-aafbb2aabb66\") " pod="openshift-monitoring/node-exporter-g6cp9" Jul 19 07:43:47.908087 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:47.907970 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/762f12f8-e846-45b9-9ba5-aafbb2aabb66-node-exporter-tls\") pod \"node-exporter-g6cp9\" (UID: \"762f12f8-e846-45b9-9ba5-aafbb2aabb66\") " pod="openshift-monitoring/node-exporter-g6cp9" Jul 19 07:43:47.908087 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:47.908009 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/762f12f8-e846-45b9-9ba5-aafbb2aabb66-sys\") pod \"node-exporter-g6cp9\" (UID: \"762f12f8-e846-45b9-9ba5-aafbb2aabb66\") " pod="openshift-monitoring/node-exporter-g6cp9" Jul 19 07:43:47.908087 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:47.908031 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/762f12f8-e846-45b9-9ba5-aafbb2aabb66-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-g6cp9\" (UID: \"762f12f8-e846-45b9-9ba5-aafbb2aabb66\") " pod="openshift-monitoring/node-exporter-g6cp9" Jul 19 07:43:47.908335 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:47.908120 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/762f12f8-e846-45b9-9ba5-aafbb2aabb66-node-exporter-textfile\") pod \"node-exporter-g6cp9\" (UID: \"762f12f8-e846-45b9-9ba5-aafbb2aabb66\") " pod="openshift-monitoring/node-exporter-g6cp9" Jul 19 07:43:47.908335 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:47.908141 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/762f12f8-e846-45b9-9ba5-aafbb2aabb66-root\") pod \"node-exporter-g6cp9\" (UID: \"762f12f8-e846-45b9-9ba5-aafbb2aabb66\") " pod="openshift-monitoring/node-exporter-g6cp9" Jul 19 07:43:47.908335 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:47.908160 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/762f12f8-e846-45b9-9ba5-aafbb2aabb66-node-exporter-accelerators-collector-config\") pod \"node-exporter-g6cp9\" (UID: \"762f12f8-e846-45b9-9ba5-aafbb2aabb66\") " pod="openshift-monitoring/node-exporter-g6cp9" Jul 19 07:43:47.965653 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:47.965608 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:48.009535 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:48.009480 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/762f12f8-e846-45b9-9ba5-aafbb2aabb66-metrics-client-ca\") pod \"node-exporter-g6cp9\" (UID: \"762f12f8-e846-45b9-9ba5-aafbb2aabb66\") " pod="openshift-monitoring/node-exporter-g6cp9" Jul 19 07:43:48.009733 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:48.009544 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/762f12f8-e846-45b9-9ba5-aafbb2aabb66-node-exporter-tls\") pod \"node-exporter-g6cp9\" (UID: \"762f12f8-e846-45b9-9ba5-aafbb2aabb66\") " pod="openshift-monitoring/node-exporter-g6cp9" Jul 19 07:43:48.009733 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:48.009578 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/762f12f8-e846-45b9-9ba5-aafbb2aabb66-sys\") pod \"node-exporter-g6cp9\" (UID: \"762f12f8-e846-45b9-9ba5-aafbb2aabb66\") " pod="openshift-monitoring/node-exporter-g6cp9" Jul 19 07:43:48.009733 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:48.009605 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/762f12f8-e846-45b9-9ba5-aafbb2aabb66-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-g6cp9\" (UID: \"762f12f8-e846-45b9-9ba5-aafbb2aabb66\") " pod="openshift-monitoring/node-exporter-g6cp9" Jul 19 07:43:48.009733 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:48.009656 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/762f12f8-e846-45b9-9ba5-aafbb2aabb66-node-exporter-textfile\") pod \"node-exporter-g6cp9\" (UID: \"762f12f8-e846-45b9-9ba5-aafbb2aabb66\") " pod="openshift-monitoring/node-exporter-g6cp9" Jul 19 07:43:48.009733 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:48.009686 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/762f12f8-e846-45b9-9ba5-aafbb2aabb66-root\") pod \"node-exporter-g6cp9\" (UID: \"762f12f8-e846-45b9-9ba5-aafbb2aabb66\") " pod="openshift-monitoring/node-exporter-g6cp9" Jul 19 07:43:48.009733 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:43:48.009686 2574 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 19 07:43:48.009733 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:48.009711 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/762f12f8-e846-45b9-9ba5-aafbb2aabb66-node-exporter-accelerators-collector-config\") pod \"node-exporter-g6cp9\" (UID: \"762f12f8-e846-45b9-9ba5-aafbb2aabb66\") " pod="openshift-monitoring/node-exporter-g6cp9" Jul 19 07:43:48.009733 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:48.009722 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/762f12f8-e846-45b9-9ba5-aafbb2aabb66-sys\") pod \"node-exporter-g6cp9\" (UID: \"762f12f8-e846-45b9-9ba5-aafbb2aabb66\") " pod="openshift-monitoring/node-exporter-g6cp9" Jul 19 07:43:48.010219 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:48.009759 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgntq\" (UniqueName: \"kubernetes.io/projected/762f12f8-e846-45b9-9ba5-aafbb2aabb66-kube-api-access-dgntq\") pod \"node-exporter-g6cp9\" (UID: \"762f12f8-e846-45b9-9ba5-aafbb2aabb66\") " pod="openshift-monitoring/node-exporter-g6cp9" Jul 19 07:43:48.010219 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:43:48.009781 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/762f12f8-e846-45b9-9ba5-aafbb2aabb66-node-exporter-tls podName:762f12f8-e846-45b9-9ba5-aafbb2aabb66 nodeName:}" failed. No retries permitted until 2026-07-19 07:43:48.509756139 +0000 UTC m=+94.864065733 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/762f12f8-e846-45b9-9ba5-aafbb2aabb66-node-exporter-tls") pod "node-exporter-g6cp9" (UID: "762f12f8-e846-45b9-9ba5-aafbb2aabb66") : secret "node-exporter-tls" not found Jul 19 07:43:48.010219 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:48.009791 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/762f12f8-e846-45b9-9ba5-aafbb2aabb66-root\") pod \"node-exporter-g6cp9\" (UID: \"762f12f8-e846-45b9-9ba5-aafbb2aabb66\") " pod="openshift-monitoring/node-exporter-g6cp9" Jul 19 07:43:48.010219 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:48.009985 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/762f12f8-e846-45b9-9ba5-aafbb2aabb66-node-exporter-wtmp\") pod \"node-exporter-g6cp9\" (UID: \"762f12f8-e846-45b9-9ba5-aafbb2aabb66\") " pod="openshift-monitoring/node-exporter-g6cp9" Jul 19 07:43:48.010219 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:48.010058 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/762f12f8-e846-45b9-9ba5-aafbb2aabb66-node-exporter-textfile\") pod \"node-exporter-g6cp9\" (UID: \"762f12f8-e846-45b9-9ba5-aafbb2aabb66\") " pod="openshift-monitoring/node-exporter-g6cp9" Jul 19 07:43:48.010219 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:48.010152 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/762f12f8-e846-45b9-9ba5-aafbb2aabb66-node-exporter-wtmp\") pod \"node-exporter-g6cp9\" (UID: \"762f12f8-e846-45b9-9ba5-aafbb2aabb66\") " pod="openshift-monitoring/node-exporter-g6cp9" Jul 19 07:43:48.010427 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:48.010222 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/762f12f8-e846-45b9-9ba5-aafbb2aabb66-metrics-client-ca\") pod \"node-exporter-g6cp9\" (UID: \"762f12f8-e846-45b9-9ba5-aafbb2aabb66\") " pod="openshift-monitoring/node-exporter-g6cp9" Jul 19 07:43:48.010427 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:48.010405 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/762f12f8-e846-45b9-9ba5-aafbb2aabb66-node-exporter-accelerators-collector-config\") pod \"node-exporter-g6cp9\" (UID: \"762f12f8-e846-45b9-9ba5-aafbb2aabb66\") " pod="openshift-monitoring/node-exporter-g6cp9" Jul 19 07:43:48.012386 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:48.012351 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/762f12f8-e846-45b9-9ba5-aafbb2aabb66-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-g6cp9\" (UID: \"762f12f8-e846-45b9-9ba5-aafbb2aabb66\") " pod="openshift-monitoring/node-exporter-g6cp9" Jul 19 07:43:48.020189 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:48.020162 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgntq\" (UniqueName: \"kubernetes.io/projected/762f12f8-e846-45b9-9ba5-aafbb2aabb66-kube-api-access-dgntq\") pod \"node-exporter-g6cp9\" (UID: \"762f12f8-e846-45b9-9ba5-aafbb2aabb66\") " pod="openshift-monitoring/node-exporter-g6cp9" Jul 19 07:43:48.066437 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:48.066344 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:48.166273 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:48.166237 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:48.241263 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:48.241221 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:48.302061 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:48.302013 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:48.513592 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:48.513546 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/762f12f8-e846-45b9-9ba5-aafbb2aabb66-node-exporter-tls\") pod \"node-exporter-g6cp9\" (UID: \"762f12f8-e846-45b9-9ba5-aafbb2aabb66\") " pod="openshift-monitoring/node-exporter-g6cp9" Jul 19 07:43:48.515845 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:48.515809 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/762f12f8-e846-45b9-9ba5-aafbb2aabb66-node-exporter-tls\") pod \"node-exporter-g6cp9\" (UID: \"762f12f8-e846-45b9-9ba5-aafbb2aabb66\") " pod="openshift-monitoring/node-exporter-g6cp9" Jul 19 07:43:48.722550 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:48.722498 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-g6cp9" Jul 19 07:43:48.736655 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:43:48.736615 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod762f12f8_e846_45b9_9ba5_aafbb2aabb66.slice/crio-32444f025318f05454dcc433ba66870fb5c1781b9fa02c6536b0ab01cdde62fe WatchSource:0}: Error finding container 32444f025318f05454dcc433ba66870fb5c1781b9fa02c6536b0ab01cdde62fe: Status 404 returned error can't find the container with id 32444f025318f05454dcc433ba66870fb5c1781b9fa02c6536b0ab01cdde62fe Jul 19 07:43:48.773727 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:48.773628 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:48.854087 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:48.854045 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:49.185424 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:49.185328 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:49.346630 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:49.346585 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-g6cp9" event={"ID":"762f12f8-e846-45b9-9ba5-aafbb2aabb66","Type":"ContainerStarted","Data":"32444f025318f05454dcc433ba66870fb5c1781b9fa02c6536b0ab01cdde62fe"} Jul 19 07:43:50.184442 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:50.184342 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:50.233972 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:50.233929 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:50.351821 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:50.351786 2574 generic.go:356] "Generic (PLEG): container finished" podID="762f12f8-e846-45b9-9ba5-aafbb2aabb66" containerID="1957dd17e05459608f7cd3c45ba98014259f443cd9572897ee82e8e0ee30b962" exitCode=0 Jul 19 07:43:50.351821 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:50.351828 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-g6cp9" event={"ID":"762f12f8-e846-45b9-9ba5-aafbb2aabb66","Type":"ContainerDied","Data":"1957dd17e05459608f7cd3c45ba98014259f443cd9572897ee82e8e0ee30b962"} Jul 19 07:43:50.703684 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:50.703563 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:50.768392 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:50.768351 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:50.911653 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:50.911617 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:51.202892 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:51.202828 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:51.358233 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:51.358200 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-g6cp9" event={"ID":"762f12f8-e846-45b9-9ba5-aafbb2aabb66","Type":"ContainerStarted","Data":"7eb033df931dac4c76744d1c7196ab491e770bf3ea977c88c4b655ebc07a7c43"} Jul 19 07:43:51.358666 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:51.358253 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-g6cp9" event={"ID":"762f12f8-e846-45b9-9ba5-aafbb2aabb66","Type":"ContainerStarted","Data":"d0679abc67903b66c290e108928705dd4f7817197f4bd8e59f6dc0686ab911af"} Jul 19 07:43:51.391265 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:51.391213 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-g6cp9" podStartSLOduration=3.242857014 podStartE2EDuration="4.391195352s" totalImagesPullingTime="1.08368971s" totalInitContainerRuntime="64.648628ms" isStatefulPod=true podCreationTimestamp="2026-07-19 07:43:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:43:51.388932339 +0000 UTC m=+97.743241943" watchObservedRunningTime="2026-07-19 07:43:51.391195352 +0000 UTC m=+97.745504953" Jul 19 07:43:52.209083 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:52.209039 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:52.259566 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:52.259506 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:52.569048 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:52.568978 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:52.668337 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:52.668298 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:52.906832 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:52.906749 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:52.955074 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:52.955030 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:53.035983 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:53.035942 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:53.970711 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:53.970665 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:54.063387 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:54.063353 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:54.165280 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:54.165241 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:55.111600 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:55.111548 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:55.205426 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:55.205396 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-lnxd5" Jul 19 07:43:55.212282 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:55.212257 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:56.308342 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:56.308308 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:56.361864 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:56.361828 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:59.248574 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:59.248507 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:43:59.348387 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:43:59.348349 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:44:00.256918 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:44:00.256877 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:44:00.359303 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:44:00.359248 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:44:01.512663 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:44:01.512624 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:44:01.560793 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:44:01.560745 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:44:04.127762 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:44:04.127724 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:44:04.231249 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:44:04.231207 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:44:07.164509 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:44:07.164468 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:44:07.265998 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:44:07.265959 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:44:07.363884 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:44:07.363832 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:44:09.326129 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:44:09.326085 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:44:09.383192 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:44:09.383150 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:44:11.112814 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:44:11.112776 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7dbdbbf55d-6mrcs" podUID="9d4e8a76-432d-4750-bb41-9d86bdb458b9" containerName="registry" containerID="cri-o://0e6fad5f1a5c54abce6529ac7d293ab6336e2c93bc356d855d4f7b685d4dbc3d" gracePeriod=30 Jul 19 07:44:11.408690 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:44:11.408662 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7dbdbbf55d-6mrcs" Jul 19 07:44:11.445647 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:44:11.445609 2574 generic.go:356] "Generic (PLEG): container finished" podID="fc2e627b-03e6-48c5-a476-00eaa505368a" containerID="76b92993c8f7f80bab9793d99b376a2cb304324741fd02581437ec18812f675b" exitCode=0 Jul 19 07:44:11.445819 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:44:11.445691 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-2hx2z" event={"ID":"fc2e627b-03e6-48c5-a476-00eaa505368a","Type":"ContainerDied","Data":"76b92993c8f7f80bab9793d99b376a2cb304324741fd02581437ec18812f675b"} Jul 19 07:44:11.446086 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:44:11.446067 2574 scope.go:117] "RemoveContainer" containerID="76b92993c8f7f80bab9793d99b376a2cb304324741fd02581437ec18812f675b" Jul 19 07:44:11.447163 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:44:11.447135 2574 generic.go:356] "Generic (PLEG): container finished" podID="9d4e8a76-432d-4750-bb41-9d86bdb458b9" containerID="0e6fad5f1a5c54abce6529ac7d293ab6336e2c93bc356d855d4f7b685d4dbc3d" exitCode=0 Jul 19 07:44:11.447269 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:44:11.447180 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7dbdbbf55d-6mrcs" Jul 19 07:44:11.447269 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:44:11.447198 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7dbdbbf55d-6mrcs" event={"ID":"9d4e8a76-432d-4750-bb41-9d86bdb458b9","Type":"ContainerDied","Data":"0e6fad5f1a5c54abce6529ac7d293ab6336e2c93bc356d855d4f7b685d4dbc3d"} Jul 19 07:44:11.447380 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:44:11.447266 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7dbdbbf55d-6mrcs" event={"ID":"9d4e8a76-432d-4750-bb41-9d86bdb458b9","Type":"ContainerDied","Data":"4aa2156f6a5ce65ef36164de8f2d2545873055e58d211a629dd9b3c8fb7f42c8"} Jul 19 07:44:11.447380 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:44:11.447293 2574 scope.go:117] "RemoveContainer" containerID="0e6fad5f1a5c54abce6529ac7d293ab6336e2c93bc356d855d4f7b685d4dbc3d" Jul 19 07:44:11.458391 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:44:11.458334 2574 scope.go:117] "RemoveContainer" containerID="0e6fad5f1a5c54abce6529ac7d293ab6336e2c93bc356d855d4f7b685d4dbc3d" Jul 19 07:44:11.458759 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:44:11.458731 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e6fad5f1a5c54abce6529ac7d293ab6336e2c93bc356d855d4f7b685d4dbc3d\": container with ID starting with 0e6fad5f1a5c54abce6529ac7d293ab6336e2c93bc356d855d4f7b685d4dbc3d not found: ID does not exist" containerID="0e6fad5f1a5c54abce6529ac7d293ab6336e2c93bc356d855d4f7b685d4dbc3d" Jul 19 07:44:11.458839 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:44:11.458770 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e6fad5f1a5c54abce6529ac7d293ab6336e2c93bc356d855d4f7b685d4dbc3d"} err="failed to get container status \"0e6fad5f1a5c54abce6529ac7d293ab6336e2c93bc356d855d4f7b685d4dbc3d\": rpc error: code = NotFound desc = could not find container \"0e6fad5f1a5c54abce6529ac7d293ab6336e2c93bc356d855d4f7b685d4dbc3d\": container with ID starting with 0e6fad5f1a5c54abce6529ac7d293ab6336e2c93bc356d855d4f7b685d4dbc3d not found: ID does not exist" Jul 19 07:44:11.517815 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:44:11.517789 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9d4e8a76-432d-4750-bb41-9d86bdb458b9-installation-pull-secrets\") pod \"9d4e8a76-432d-4750-bb41-9d86bdb458b9\" (UID: \"9d4e8a76-432d-4750-bb41-9d86bdb458b9\") " Jul 19 07:44:11.517989 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:44:11.517823 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9d4e8a76-432d-4750-bb41-9d86bdb458b9-image-registry-private-configuration\") pod \"9d4e8a76-432d-4750-bb41-9d86bdb458b9\" (UID: \"9d4e8a76-432d-4750-bb41-9d86bdb458b9\") " Jul 19 07:44:11.517989 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:44:11.517868 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5ntqn\" (UniqueName: \"kubernetes.io/projected/9d4e8a76-432d-4750-bb41-9d86bdb458b9-kube-api-access-5ntqn\") pod \"9d4e8a76-432d-4750-bb41-9d86bdb458b9\" (UID: \"9d4e8a76-432d-4750-bb41-9d86bdb458b9\") " Jul 19 07:44:11.517989 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:44:11.517918 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9d4e8a76-432d-4750-bb41-9d86bdb458b9-ca-trust-extracted\") pod \"9d4e8a76-432d-4750-bb41-9d86bdb458b9\" (UID: \"9d4e8a76-432d-4750-bb41-9d86bdb458b9\") " Jul 19 07:44:11.517989 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:44:11.517946 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9d4e8a76-432d-4750-bb41-9d86bdb458b9-registry-certificates\") pod \"9d4e8a76-432d-4750-bb41-9d86bdb458b9\" (UID: \"9d4e8a76-432d-4750-bb41-9d86bdb458b9\") " Jul 19 07:44:11.518196 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:44:11.518069 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9d4e8a76-432d-4750-bb41-9d86bdb458b9-registry-tls\") pod \"9d4e8a76-432d-4750-bb41-9d86bdb458b9\" (UID: \"9d4e8a76-432d-4750-bb41-9d86bdb458b9\") " Jul 19 07:44:11.518196 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:44:11.518114 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9d4e8a76-432d-4750-bb41-9d86bdb458b9-bound-sa-token\") pod \"9d4e8a76-432d-4750-bb41-9d86bdb458b9\" (UID: \"9d4e8a76-432d-4750-bb41-9d86bdb458b9\") " Jul 19 07:44:11.518196 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:44:11.518188 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9d4e8a76-432d-4750-bb41-9d86bdb458b9-trusted-ca\") pod \"9d4e8a76-432d-4750-bb41-9d86bdb458b9\" (UID: \"9d4e8a76-432d-4750-bb41-9d86bdb458b9\") " Jul 19 07:44:11.518387 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:44:11.518364 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d4e8a76-432d-4750-bb41-9d86bdb458b9-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "9d4e8a76-432d-4750-bb41-9d86bdb458b9" (UID: "9d4e8a76-432d-4750-bb41-9d86bdb458b9"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 07:44:11.518534 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:44:11.518504 2574 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9d4e8a76-432d-4750-bb41-9d86bdb458b9-registry-certificates\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 07:44:11.518819 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:44:11.518778 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d4e8a76-432d-4750-bb41-9d86bdb458b9-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "9d4e8a76-432d-4750-bb41-9d86bdb458b9" (UID: "9d4e8a76-432d-4750-bb41-9d86bdb458b9"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 07:44:11.520496 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:44:11.520462 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d4e8a76-432d-4750-bb41-9d86bdb458b9-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "9d4e8a76-432d-4750-bb41-9d86bdb458b9" (UID: "9d4e8a76-432d-4750-bb41-9d86bdb458b9"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 07:44:11.520638 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:44:11.520605 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d4e8a76-432d-4750-bb41-9d86bdb458b9-kube-api-access-5ntqn" (OuterVolumeSpecName: "kube-api-access-5ntqn") pod "9d4e8a76-432d-4750-bb41-9d86bdb458b9" (UID: "9d4e8a76-432d-4750-bb41-9d86bdb458b9"). InnerVolumeSpecName "kube-api-access-5ntqn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 07:44:11.520638 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:44:11.520624 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d4e8a76-432d-4750-bb41-9d86bdb458b9-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "9d4e8a76-432d-4750-bb41-9d86bdb458b9" (UID: "9d4e8a76-432d-4750-bb41-9d86bdb458b9"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 07:44:11.520767 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:44:11.520699 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d4e8a76-432d-4750-bb41-9d86bdb458b9-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "9d4e8a76-432d-4750-bb41-9d86bdb458b9" (UID: "9d4e8a76-432d-4750-bb41-9d86bdb458b9"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 07:44:11.520875 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:44:11.520853 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d4e8a76-432d-4750-bb41-9d86bdb458b9-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "9d4e8a76-432d-4750-bb41-9d86bdb458b9" (UID: "9d4e8a76-432d-4750-bb41-9d86bdb458b9"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 07:44:11.526341 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:44:11.526318 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d4e8a76-432d-4750-bb41-9d86bdb458b9-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "9d4e8a76-432d-4750-bb41-9d86bdb458b9" (UID: "9d4e8a76-432d-4750-bb41-9d86bdb458b9"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 07:44:11.619645 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:44:11.619545 2574 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9d4e8a76-432d-4750-bb41-9d86bdb458b9-installation-pull-secrets\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 07:44:11.619795 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:44:11.619693 2574 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9d4e8a76-432d-4750-bb41-9d86bdb458b9-image-registry-private-configuration\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 07:44:11.619795 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:44:11.619726 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ntqn\" (UniqueName: \"kubernetes.io/projected/9d4e8a76-432d-4750-bb41-9d86bdb458b9-kube-api-access-5ntqn\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 07:44:11.619795 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:44:11.619742 2574 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9d4e8a76-432d-4750-bb41-9d86bdb458b9-ca-trust-extracted\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 07:44:11.619795 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:44:11.619757 2574 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9d4e8a76-432d-4750-bb41-9d86bdb458b9-registry-tls\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 07:44:11.619795 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:44:11.619772 2574 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9d4e8a76-432d-4750-bb41-9d86bdb458b9-bound-sa-token\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 07:44:11.619795 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:44:11.619785 2574 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9d4e8a76-432d-4750-bb41-9d86bdb458b9-trusted-ca\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 07:44:11.769091 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:44:11.769051 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7dbdbbf55d-6mrcs"] Jul 19 07:44:11.773873 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:44:11.773845 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7dbdbbf55d-6mrcs"] Jul 19 07:44:12.453042 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:44:12.453005 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-2hx2z" event={"ID":"fc2e627b-03e6-48c5-a476-00eaa505368a","Type":"ContainerStarted","Data":"9135721f28568151c61fe17349f56c163ddbcf985e2dea2d60f6a25ae0c5ffdd"} Jul 19 07:44:12.558456 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:44:12.558425 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d4e8a76-432d-4750-bb41-9d86bdb458b9" path="/var/lib/kubelet/pods/9d4e8a76-432d-4750-bb41-9d86bdb458b9/volumes" Jul 19 07:44:16.472541 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:44:16.472481 2574 generic.go:356] "Generic (PLEG): container finished" podID="01c43e18-77be-4ec4-88b3-6a5662c66917" containerID="68f610967d8fe88409a5abdc14d7a5b21653de1a7ba2133eb862c98f9e53ad9c" exitCode=0 Jul 19 07:44:16.472976 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:44:16.472564 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-hsd5x" event={"ID":"01c43e18-77be-4ec4-88b3-6a5662c66917","Type":"ContainerDied","Data":"68f610967d8fe88409a5abdc14d7a5b21653de1a7ba2133eb862c98f9e53ad9c"} Jul 19 07:44:16.472976 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:44:16.472911 2574 scope.go:117] "RemoveContainer" containerID="68f610967d8fe88409a5abdc14d7a5b21653de1a7ba2133eb862c98f9e53ad9c" Jul 19 07:44:17.477911 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:44:17.477870 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-hsd5x" event={"ID":"01c43e18-77be-4ec4-88b3-6a5662c66917","Type":"ContainerStarted","Data":"51a201961b1b38e1e55f9fccb69f25d3f49ba66ba5be7bd3b095738839971666"} Jul 19 07:44:22.202209 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:44:22.202175 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:44:22.248902 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:44:22.248869 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:44:29.480798 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:44:29.480749 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:44:29.581576 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:44:29.581535 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:44:29.629096 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:44:29.629047 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:44:31.531279 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:44:31.531241 2574 generic.go:356] "Generic (PLEG): container finished" podID="0adaffaf-8ab7-4966-abd1-5bbd8a0ba882" containerID="8b89562136e75538ed3be5c2adae6b624ff9573d9b5a4b1be785850eb9b5cb98" exitCode=0 Jul 19 07:44:31.531681 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:44:31.531310 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-xr7x7" event={"ID":"0adaffaf-8ab7-4966-abd1-5bbd8a0ba882","Type":"ContainerDied","Data":"8b89562136e75538ed3be5c2adae6b624ff9573d9b5a4b1be785850eb9b5cb98"} Jul 19 07:44:31.531724 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:44:31.531684 2574 scope.go:117] "RemoveContainer" containerID="8b89562136e75538ed3be5c2adae6b624ff9573d9b5a4b1be785850eb9b5cb98" Jul 19 07:44:32.537548 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:44:32.537499 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-xr7x7" event={"ID":"0adaffaf-8ab7-4966-abd1-5bbd8a0ba882","Type":"ContainerStarted","Data":"2e28b1d1c8ba200ac80195942719494e9a736abfca9d730ccb9cad7a168ea4da"} Jul 19 07:44:37.198990 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:44:37.198953 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:44:37.248302 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:44:37.248267 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:45:06.957012 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:45:06.956912 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:45:07.158125 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:45:07.158092 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:45:17.055266 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:45:17.055223 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:45:28.497893 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:45:28.497857 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:45:28.545977 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:45:28.545941 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:45:28.597659 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:45:28.597620 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:46:02.270801 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:46:02.270763 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:46:02.317089 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:46:02.317048 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:46:22.209486 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:46:22.209452 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:46:22.259906 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:46:22.259854 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:46:29.505929 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:46:29.505892 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:46:29.562448 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:46:29.562406 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:46:29.641594 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:46:29.641560 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:46:29.689486 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:46:29.689454 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:46:30.240030 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:46:30.239988 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:46:30.288412 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:46:30.288373 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:46:34.073048 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:46:34.073012 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:46:34.121892 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:46:34.121856 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:46:34.378030 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:46:34.377937 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:46:37.204904 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:46:37.204809 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:46:37.305134 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:46:37.305098 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:46:37.357749 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:46:37.357715 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:46:40.708472 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:46:40.708437 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:46:40.756403 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:46:40.756364 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:46:40.811580 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:46:40.811548 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:46:40.863798 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:46:40.863764 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:47:14.460658 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:47:14.460623 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wmhfd_dd68d31d-f763-433a-867b-65b12eabac76/console-operator/1.log" Jul 19 07:47:14.467533 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:47:14.467493 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wmhfd_dd68d31d-f763-433a-867b-65b12eabac76/console-operator/1.log" Jul 19 07:47:14.471080 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:47:14.471052 2574 kubelet.go:1648] "Image garbage collection succeeded" Jul 19 07:47:32.959740 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:47:32.959698 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:47:34.013301 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:47:34.013246 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:47:54.515237 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:47:54.515195 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:47:55.332995 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:47:55.332957 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:47:58.116033 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:47:58.115989 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:47:58.216320 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:47:58.216280 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:47:59.277331 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:47:59.277290 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:47:59.372183 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:47:59.372144 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:47:59.998889 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:47:59.998845 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:48:00.096128 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:00.096089 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:48:01.994062 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:01.993986 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:48:02.393313 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:02.393226 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:48:04.914247 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:04.914200 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:48:04.967046 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:04.967012 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:48:08.782999 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:08.782907 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:48:08.831504 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:08.831468 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:48:10.619615 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:10.619573 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:48:10.926222 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:10.926117 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:48:11.026270 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:11.026227 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:48:16.695417 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:16.695363 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:48:16.785793 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:16.785757 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:48:18.460610 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:18.460577 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:48:33.338848 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:33.338803 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:48:33.450529 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:33.450496 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:48:33.949189 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:33.949152 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:48:34.153575 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:34.153541 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:48:34.277421 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:34.277378 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:48:34.324793 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:34.324755 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-789467ff8c-j7mlr"] Jul 19 07:48:34.325078 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:34.325064 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d4e8a76-432d-4750-bb41-9d86bdb458b9" containerName="registry" Jul 19 07:48:34.325078 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:34.325079 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d4e8a76-432d-4750-bb41-9d86bdb458b9" containerName="registry" Jul 19 07:48:34.325176 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:34.325139 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d4e8a76-432d-4750-bb41-9d86bdb458b9" containerName="registry" Jul 19 07:48:34.328249 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:34.328227 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-789467ff8c-j7mlr" Jul 19 07:48:34.331287 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:34.331265 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 19 07:48:34.334501 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:34.334474 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 19 07:48:34.334639 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:34.334625 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-8c278\"" Jul 19 07:48:34.334759 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:34.334744 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 19 07:48:34.335359 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:34.335339 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 19 07:48:34.349769 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:34.349740 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-789467ff8c-j7mlr"] Jul 19 07:48:34.466974 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:34.466937 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:48:34.480230 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:34.480198 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6e217fee-5171-40e1-82fb-155e4a0431d1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-789467ff8c-j7mlr\" (UID: \"6e217fee-5171-40e1-82fb-155e4a0431d1\") " pod="opendatahub/opendatahub-operator-controller-manager-789467ff8c-j7mlr" Jul 19 07:48:34.480382 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:34.480250 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csb6g\" (UniqueName: \"kubernetes.io/projected/6e217fee-5171-40e1-82fb-155e4a0431d1-kube-api-access-csb6g\") pod \"opendatahub-operator-controller-manager-789467ff8c-j7mlr\" (UID: \"6e217fee-5171-40e1-82fb-155e4a0431d1\") " pod="opendatahub/opendatahub-operator-controller-manager-789467ff8c-j7mlr" Jul 19 07:48:34.480382 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:34.480342 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6e217fee-5171-40e1-82fb-155e4a0431d1-webhook-cert\") pod \"opendatahub-operator-controller-manager-789467ff8c-j7mlr\" (UID: \"6e217fee-5171-40e1-82fb-155e4a0431d1\") " pod="opendatahub/opendatahub-operator-controller-manager-789467ff8c-j7mlr" Jul 19 07:48:34.480478 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:34.480406 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6e217fee-5171-40e1-82fb-155e4a0431d1-manifests\") pod \"opendatahub-operator-controller-manager-789467ff8c-j7mlr\" (UID: \"6e217fee-5171-40e1-82fb-155e4a0431d1\") " pod="opendatahub/opendatahub-operator-controller-manager-789467ff8c-j7mlr" Jul 19 07:48:34.480478 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:34.480465 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6e217fee-5171-40e1-82fb-155e4a0431d1-tmp\") pod \"opendatahub-operator-controller-manager-789467ff8c-j7mlr\" (UID: \"6e217fee-5171-40e1-82fb-155e4a0431d1\") " pod="opendatahub/opendatahub-operator-controller-manager-789467ff8c-j7mlr" Jul 19 07:48:34.564510 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:34.564424 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:48:34.581656 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:34.581627 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6e217fee-5171-40e1-82fb-155e4a0431d1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-789467ff8c-j7mlr\" (UID: \"6e217fee-5171-40e1-82fb-155e4a0431d1\") " pod="opendatahub/opendatahub-operator-controller-manager-789467ff8c-j7mlr" Jul 19 07:48:34.581815 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:34.581662 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csb6g\" (UniqueName: \"kubernetes.io/projected/6e217fee-5171-40e1-82fb-155e4a0431d1-kube-api-access-csb6g\") pod \"opendatahub-operator-controller-manager-789467ff8c-j7mlr\" (UID: \"6e217fee-5171-40e1-82fb-155e4a0431d1\") " pod="opendatahub/opendatahub-operator-controller-manager-789467ff8c-j7mlr" Jul 19 07:48:34.581815 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:34.581689 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6e217fee-5171-40e1-82fb-155e4a0431d1-webhook-cert\") pod \"opendatahub-operator-controller-manager-789467ff8c-j7mlr\" (UID: \"6e217fee-5171-40e1-82fb-155e4a0431d1\") " pod="opendatahub/opendatahub-operator-controller-manager-789467ff8c-j7mlr" Jul 19 07:48:34.581815 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:34.581733 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6e217fee-5171-40e1-82fb-155e4a0431d1-manifests\") pod \"opendatahub-operator-controller-manager-789467ff8c-j7mlr\" (UID: \"6e217fee-5171-40e1-82fb-155e4a0431d1\") " pod="opendatahub/opendatahub-operator-controller-manager-789467ff8c-j7mlr" Jul 19 07:48:34.581815 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:34.581760 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6e217fee-5171-40e1-82fb-155e4a0431d1-tmp\") pod \"opendatahub-operator-controller-manager-789467ff8c-j7mlr\" (UID: \"6e217fee-5171-40e1-82fb-155e4a0431d1\") " pod="opendatahub/opendatahub-operator-controller-manager-789467ff8c-j7mlr" Jul 19 07:48:34.582266 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:34.582246 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6e217fee-5171-40e1-82fb-155e4a0431d1-manifests\") pod \"opendatahub-operator-controller-manager-789467ff8c-j7mlr\" (UID: \"6e217fee-5171-40e1-82fb-155e4a0431d1\") " pod="opendatahub/opendatahub-operator-controller-manager-789467ff8c-j7mlr" Jul 19 07:48:34.584149 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:34.584122 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6e217fee-5171-40e1-82fb-155e4a0431d1-tmp\") pod \"opendatahub-operator-controller-manager-789467ff8c-j7mlr\" (UID: \"6e217fee-5171-40e1-82fb-155e4a0431d1\") " pod="opendatahub/opendatahub-operator-controller-manager-789467ff8c-j7mlr" Jul 19 07:48:34.584249 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:34.584183 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6e217fee-5171-40e1-82fb-155e4a0431d1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-789467ff8c-j7mlr\" (UID: \"6e217fee-5171-40e1-82fb-155e4a0431d1\") " pod="opendatahub/opendatahub-operator-controller-manager-789467ff8c-j7mlr" Jul 19 07:48:34.584308 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:34.584269 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6e217fee-5171-40e1-82fb-155e4a0431d1-webhook-cert\") pod \"opendatahub-operator-controller-manager-789467ff8c-j7mlr\" (UID: \"6e217fee-5171-40e1-82fb-155e4a0431d1\") " pod="opendatahub/opendatahub-operator-controller-manager-789467ff8c-j7mlr" Jul 19 07:48:34.600498 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:34.600473 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-csb6g\" (UniqueName: \"kubernetes.io/projected/6e217fee-5171-40e1-82fb-155e4a0431d1-kube-api-access-csb6g\") pod \"opendatahub-operator-controller-manager-789467ff8c-j7mlr\" (UID: \"6e217fee-5171-40e1-82fb-155e4a0431d1\") " pod="opendatahub/opendatahub-operator-controller-manager-789467ff8c-j7mlr" Jul 19 07:48:34.638595 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:34.638547 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-789467ff8c-j7mlr" Jul 19 07:48:34.668078 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:34.667960 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:48:34.756480 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:34.756442 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:48:34.812470 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:34.808028 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-789467ff8c-j7mlr"] Jul 19 07:48:34.812470 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:48:34.812430 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e217fee_5171_40e1_82fb_155e4a0431d1.slice/crio-e8ed56fc2b9e75280c1b4400363af433d3a536458fe5cc3aab98aac5e9441497 WatchSource:0}: Error finding container e8ed56fc2b9e75280c1b4400363af433d3a536458fe5cc3aab98aac5e9441497: Status 404 returned error can't find the container with id e8ed56fc2b9e75280c1b4400363af433d3a536458fe5cc3aab98aac5e9441497 Jul 19 07:48:34.813396 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:34.813375 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 19 07:48:34.874616 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:34.874572 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:48:34.971355 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:34.971313 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:48:35.483464 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:35.483431 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-789467ff8c-j7mlr" event={"ID":"6e217fee-5171-40e1-82fb-155e4a0431d1","Type":"ContainerStarted","Data":"e8ed56fc2b9e75280c1b4400363af433d3a536458fe5cc3aab98aac5e9441497"} Jul 19 07:48:38.499468 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:38.499428 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-789467ff8c-j7mlr" event={"ID":"6e217fee-5171-40e1-82fb-155e4a0431d1","Type":"ContainerStarted","Data":"01c3b0c2bc53809b8f8aad8f1d19fdc9bf2612c6da8f950d8f1d5a30d84392ac"} Jul 19 07:48:39.505100 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:39.505060 2574 generic.go:356] "Generic (PLEG): container finished" podID="6e217fee-5171-40e1-82fb-155e4a0431d1" containerID="01c3b0c2bc53809b8f8aad8f1d19fdc9bf2612c6da8f950d8f1d5a30d84392ac" exitCode=0 Jul 19 07:48:39.505503 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:39.505151 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-789467ff8c-j7mlr" event={"ID":"6e217fee-5171-40e1-82fb-155e4a0431d1","Type":"ContainerDied","Data":"01c3b0c2bc53809b8f8aad8f1d19fdc9bf2612c6da8f950d8f1d5a30d84392ac"} Jul 19 07:48:40.511227 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:40.511193 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-789467ff8c-j7mlr" event={"ID":"6e217fee-5171-40e1-82fb-155e4a0431d1","Type":"ContainerStarted","Data":"3d2f4b6e38aa1b931f6bbd06fdbf6e4a7f729fd034570a1d5c888f8c2b2d88dc"} Jul 19 07:48:40.511668 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:40.511341 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-789467ff8c-j7mlr" Jul 19 07:48:40.533019 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:40.532955 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-789467ff8c-j7mlr" podStartSLOduration=2.288012927 podStartE2EDuration="6.532932793s" totalImagesPullingTime="3.173878369s" totalInitContainerRuntime="1.071041497s" isStatefulPod=false podCreationTimestamp="2026-07-19 07:48:34 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:48:40.530314515 +0000 UTC m=+386.884624115" watchObservedRunningTime="2026-07-19 07:48:40.532932793 +0000 UTC m=+386.887242397" Jul 19 07:48:49.963132 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:49.963090 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:48:50.019089 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:50.019046 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:48:50.385810 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:50.385717 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:48:50.440512 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:50.440475 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:48:51.517722 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:51.517690 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-789467ff8c-j7mlr" Jul 19 07:48:54.078500 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:54.078461 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:48:54.222306 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:54.222269 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:48:54.743498 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:54.743457 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-56f59f876d-qqxn9"] Jul 19 07:48:54.746122 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:54.746083 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:48:54.756943 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:54.756906 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-56f59f876d-qqxn9" Jul 19 07:48:54.759489 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:54.759462 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-56f59f876d-qqxn9"] Jul 19 07:48:54.759720 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:54.759699 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 19 07:48:54.759720 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:54.759715 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 19 07:48:54.856227 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:54.856190 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c5887e0-25ef-48da-a06c-a1874a64e2c7-tls-certs\") pod \"kube-auth-proxy-56f59f876d-qqxn9\" (UID: \"1c5887e0-25ef-48da-a06c-a1874a64e2c7\") " pod="openshift-ingress/kube-auth-proxy-56f59f876d-qqxn9" Jul 19 07:48:54.856437 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:54.856274 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45sgx\" (UniqueName: \"kubernetes.io/projected/1c5887e0-25ef-48da-a06c-a1874a64e2c7-kube-api-access-45sgx\") pod \"kube-auth-proxy-56f59f876d-qqxn9\" (UID: \"1c5887e0-25ef-48da-a06c-a1874a64e2c7\") " pod="openshift-ingress/kube-auth-proxy-56f59f876d-qqxn9" Jul 19 07:48:54.856437 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:54.856302 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1c5887e0-25ef-48da-a06c-a1874a64e2c7-tmp\") pod \"kube-auth-proxy-56f59f876d-qqxn9\" (UID: \"1c5887e0-25ef-48da-a06c-a1874a64e2c7\") " pod="openshift-ingress/kube-auth-proxy-56f59f876d-qqxn9" Jul 19 07:48:54.934095 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:54.934056 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-85fwf"] Jul 19 07:48:54.938120 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:54.938093 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-85fwf" Jul 19 07:48:54.942412 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:54.942382 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 19 07:48:54.942756 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:54.942714 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-9btdk\"" Jul 19 07:48:54.970929 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:54.964021 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-85fwf"] Jul 19 07:48:54.970929 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:54.966043 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45sgx\" (UniqueName: \"kubernetes.io/projected/1c5887e0-25ef-48da-a06c-a1874a64e2c7-kube-api-access-45sgx\") pod \"kube-auth-proxy-56f59f876d-qqxn9\" (UID: \"1c5887e0-25ef-48da-a06c-a1874a64e2c7\") " pod="openshift-ingress/kube-auth-proxy-56f59f876d-qqxn9" Jul 19 07:48:54.970929 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:54.966099 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1c5887e0-25ef-48da-a06c-a1874a64e2c7-tmp\") pod \"kube-auth-proxy-56f59f876d-qqxn9\" (UID: \"1c5887e0-25ef-48da-a06c-a1874a64e2c7\") " pod="openshift-ingress/kube-auth-proxy-56f59f876d-qqxn9" Jul 19 07:48:54.970929 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:54.966135 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c5887e0-25ef-48da-a06c-a1874a64e2c7-tls-certs\") pod \"kube-auth-proxy-56f59f876d-qqxn9\" (UID: \"1c5887e0-25ef-48da-a06c-a1874a64e2c7\") " pod="openshift-ingress/kube-auth-proxy-56f59f876d-qqxn9" Jul 19 07:48:54.970929 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:48:54.966312 2574 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Jul 19 07:48:54.970929 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:48:54.966389 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c5887e0-25ef-48da-a06c-a1874a64e2c7-tls-certs podName:1c5887e0-25ef-48da-a06c-a1874a64e2c7 nodeName:}" failed. No retries permitted until 2026-07-19 07:48:55.466360906 +0000 UTC m=+401.820670504 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/1c5887e0-25ef-48da-a06c-a1874a64e2c7-tls-certs") pod "kube-auth-proxy-56f59f876d-qqxn9" (UID: "1c5887e0-25ef-48da-a06c-a1874a64e2c7") : secret "kube-auth-proxy-tls" not found Jul 19 07:48:54.970929 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:54.968660 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:48:54.970929 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:54.970031 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1c5887e0-25ef-48da-a06c-a1874a64e2c7-tmp\") pod \"kube-auth-proxy-56f59f876d-qqxn9\" (UID: \"1c5887e0-25ef-48da-a06c-a1874a64e2c7\") " pod="openshift-ingress/kube-auth-proxy-56f59f876d-qqxn9" Jul 19 07:48:54.984307 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:54.984280 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45sgx\" (UniqueName: \"kubernetes.io/projected/1c5887e0-25ef-48da-a06c-a1874a64e2c7-kube-api-access-45sgx\") pod \"kube-auth-proxy-56f59f876d-qqxn9\" (UID: \"1c5887e0-25ef-48da-a06c-a1874a64e2c7\") " pod="openshift-ingress/kube-auth-proxy-56f59f876d-qqxn9" Jul 19 07:48:55.067188 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:55.066935 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tg6z\" (UniqueName: \"kubernetes.io/projected/d389a635-9a6d-42d5-b70c-b78a0898b929-kube-api-access-5tg6z\") pod \"odh-model-controller-7998c67dc4-85fwf\" (UID: \"d389a635-9a6d-42d5-b70c-b78a0898b929\") " pod="opendatahub/odh-model-controller-7998c67dc4-85fwf" Jul 19 07:48:55.067188 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:55.067014 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d389a635-9a6d-42d5-b70c-b78a0898b929-cert\") pod \"odh-model-controller-7998c67dc4-85fwf\" (UID: \"d389a635-9a6d-42d5-b70c-b78a0898b929\") " pod="opendatahub/odh-model-controller-7998c67dc4-85fwf" Jul 19 07:48:55.067926 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:55.067712 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:48:55.162125 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:55.162077 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:48:55.168063 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:55.168030 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d389a635-9a6d-42d5-b70c-b78a0898b929-cert\") pod \"odh-model-controller-7998c67dc4-85fwf\" (UID: \"d389a635-9a6d-42d5-b70c-b78a0898b929\") " pod="opendatahub/odh-model-controller-7998c67dc4-85fwf" Jul 19 07:48:55.168250 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:55.168157 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tg6z\" (UniqueName: \"kubernetes.io/projected/d389a635-9a6d-42d5-b70c-b78a0898b929-kube-api-access-5tg6z\") pod \"odh-model-controller-7998c67dc4-85fwf\" (UID: \"d389a635-9a6d-42d5-b70c-b78a0898b929\") " pod="opendatahub/odh-model-controller-7998c67dc4-85fwf" Jul 19 07:48:55.170608 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:55.170576 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d389a635-9a6d-42d5-b70c-b78a0898b929-cert\") pod \"odh-model-controller-7998c67dc4-85fwf\" (UID: \"d389a635-9a6d-42d5-b70c-b78a0898b929\") " pod="opendatahub/odh-model-controller-7998c67dc4-85fwf" Jul 19 07:48:55.176639 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:55.176612 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tg6z\" (UniqueName: \"kubernetes.io/projected/d389a635-9a6d-42d5-b70c-b78a0898b929-kube-api-access-5tg6z\") pod \"odh-model-controller-7998c67dc4-85fwf\" (UID: \"d389a635-9a6d-42d5-b70c-b78a0898b929\") " pod="opendatahub/odh-model-controller-7998c67dc4-85fwf" Jul 19 07:48:55.265213 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:55.265178 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-85fwf" Jul 19 07:48:55.268404 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:55.268366 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:48:55.364036 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:55.363834 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:48:55.408163 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:55.408137 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-85fwf"] Jul 19 07:48:55.412294 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:48:55.412263 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd389a635_9a6d_42d5_b70c_b78a0898b929.slice/crio-756e8a3f473147f11e09e0f18fbbe2c5b7d62bc0add52960981da98e3a6c6c63 WatchSource:0}: Error finding container 756e8a3f473147f11e09e0f18fbbe2c5b7d62bc0add52960981da98e3a6c6c63: Status 404 returned error can't find the container with id 756e8a3f473147f11e09e0f18fbbe2c5b7d62bc0add52960981da98e3a6c6c63 Jul 19 07:48:55.470764 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:55.470707 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c5887e0-25ef-48da-a06c-a1874a64e2c7-tls-certs\") pod \"kube-auth-proxy-56f59f876d-qqxn9\" (UID: \"1c5887e0-25ef-48da-a06c-a1874a64e2c7\") " pod="openshift-ingress/kube-auth-proxy-56f59f876d-qqxn9" Jul 19 07:48:55.473246 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:55.473216 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c5887e0-25ef-48da-a06c-a1874a64e2c7-tls-certs\") pod \"kube-auth-proxy-56f59f876d-qqxn9\" (UID: \"1c5887e0-25ef-48da-a06c-a1874a64e2c7\") " pod="openshift-ingress/kube-auth-proxy-56f59f876d-qqxn9" Jul 19 07:48:55.577339 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:55.577304 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-85fwf" event={"ID":"d389a635-9a6d-42d5-b70c-b78a0898b929","Type":"ContainerStarted","Data":"756e8a3f473147f11e09e0f18fbbe2c5b7d62bc0add52960981da98e3a6c6c63"} Jul 19 07:48:55.669751 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:55.669646 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-56f59f876d-qqxn9" Jul 19 07:48:55.727662 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:55.727420 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:48:55.823688 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:55.823656 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-56f59f876d-qqxn9"] Jul 19 07:48:55.825855 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:48:55.825818 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c5887e0_25ef_48da_a06c_a1874a64e2c7.slice/crio-50c3bc1e6a0ab87d36ba7c37961fb8aaacf793a114d11b4f8f4b4d22b17c43b3 WatchSource:0}: Error finding container 50c3bc1e6a0ab87d36ba7c37961fb8aaacf793a114d11b4f8f4b4d22b17c43b3: Status 404 returned error can't find the container with id 50c3bc1e6a0ab87d36ba7c37961fb8aaacf793a114d11b4f8f4b4d22b17c43b3 Jul 19 07:48:55.855599 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:55.855564 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-lvdn2"] Jul 19 07:48:55.860664 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:55.860630 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-lvdn2" Jul 19 07:48:55.863331 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:55.863304 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 19 07:48:55.863488 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:55.863316 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-kgfxr\"" Jul 19 07:48:55.868020 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:55.867966 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-lvdn2"] Jul 19 07:48:55.893072 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:55.893039 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-7fhcg"] Jul 19 07:48:55.896367 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:55.896349 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-7fhcg" Jul 19 07:48:55.899046 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:55.899020 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 19 07:48:55.899184 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:55.899072 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 19 07:48:55.899184 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:55.899088 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 19 07:48:55.899184 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:55.899099 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-lnp6j\"" Jul 19 07:48:55.899184 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:55.899076 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 19 07:48:55.908051 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:55.907973 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:48:55.909002 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:55.908979 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-7fhcg"] Jul 19 07:48:55.972646 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:55.972588 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:48:55.975658 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:55.975621 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/27499250-8f4d-478c-945c-8b3512bd8209-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-7fhcg\" (UID: \"27499250-8f4d-478c-945c-8b3512bd8209\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-7fhcg" Jul 19 07:48:55.975658 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:55.975662 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6q4c\" (UniqueName: \"kubernetes.io/projected/27499250-8f4d-478c-945c-8b3512bd8209-kube-api-access-s6q4c\") pod \"istiod-openshift-gateway-7574459656-7fhcg\" (UID: \"27499250-8f4d-478c-945c-8b3512bd8209\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-7fhcg" Jul 19 07:48:55.975658 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:55.975717 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/27499250-8f4d-478c-945c-8b3512bd8209-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-7fhcg\" (UID: \"27499250-8f4d-478c-945c-8b3512bd8209\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-7fhcg" Jul 19 07:48:55.975658 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:55.975761 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/27499250-8f4d-478c-945c-8b3512bd8209-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-7fhcg\" (UID: \"27499250-8f4d-478c-945c-8b3512bd8209\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-7fhcg" Jul 19 07:48:55.975658 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:55.975800 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dm9fp\" (UniqueName: \"kubernetes.io/projected/a402f5d0-4bcc-45be-9d79-13c7e577d3c3-kube-api-access-dm9fp\") pod \"model-serving-api-54d89bd5fc-lvdn2\" (UID: \"a402f5d0-4bcc-45be-9d79-13c7e577d3c3\") " pod="opendatahub/model-serving-api-54d89bd5fc-lvdn2" Jul 19 07:48:55.975658 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:55.975827 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/27499250-8f4d-478c-945c-8b3512bd8209-local-certs\") pod \"istiod-openshift-gateway-7574459656-7fhcg\" (UID: \"27499250-8f4d-478c-945c-8b3512bd8209\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-7fhcg" Jul 19 07:48:55.977046 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:55.975864 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a402f5d0-4bcc-45be-9d79-13c7e577d3c3-tls-certs\") pod \"model-serving-api-54d89bd5fc-lvdn2\" (UID: \"a402f5d0-4bcc-45be-9d79-13c7e577d3c3\") " pod="opendatahub/model-serving-api-54d89bd5fc-lvdn2" Jul 19 07:48:55.977046 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:55.975943 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27499250-8f4d-478c-945c-8b3512bd8209-istio-token\") pod \"istiod-openshift-gateway-7574459656-7fhcg\" (UID: \"27499250-8f4d-478c-945c-8b3512bd8209\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-7fhcg" Jul 19 07:48:55.977046 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:55.975965 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/27499250-8f4d-478c-945c-8b3512bd8209-cacerts\") pod \"istiod-openshift-gateway-7574459656-7fhcg\" (UID: \"27499250-8f4d-478c-945c-8b3512bd8209\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-7fhcg" Jul 19 07:48:56.076847 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:56.076806 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/27499250-8f4d-478c-945c-8b3512bd8209-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-7fhcg\" (UID: \"27499250-8f4d-478c-945c-8b3512bd8209\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-7fhcg" Jul 19 07:48:56.077045 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:56.076863 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm9fp\" (UniqueName: \"kubernetes.io/projected/a402f5d0-4bcc-45be-9d79-13c7e577d3c3-kube-api-access-dm9fp\") pod \"model-serving-api-54d89bd5fc-lvdn2\" (UID: \"a402f5d0-4bcc-45be-9d79-13c7e577d3c3\") " pod="opendatahub/model-serving-api-54d89bd5fc-lvdn2" Jul 19 07:48:56.077045 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:56.076891 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/27499250-8f4d-478c-945c-8b3512bd8209-local-certs\") pod \"istiod-openshift-gateway-7574459656-7fhcg\" (UID: \"27499250-8f4d-478c-945c-8b3512bd8209\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-7fhcg" Jul 19 07:48:56.077045 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:56.076931 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a402f5d0-4bcc-45be-9d79-13c7e577d3c3-tls-certs\") pod \"model-serving-api-54d89bd5fc-lvdn2\" (UID: \"a402f5d0-4bcc-45be-9d79-13c7e577d3c3\") " pod="opendatahub/model-serving-api-54d89bd5fc-lvdn2" Jul 19 07:48:56.077045 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:56.076986 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27499250-8f4d-478c-945c-8b3512bd8209-istio-token\") pod \"istiod-openshift-gateway-7574459656-7fhcg\" (UID: \"27499250-8f4d-478c-945c-8b3512bd8209\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-7fhcg" Jul 19 07:48:56.077045 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:56.077008 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/27499250-8f4d-478c-945c-8b3512bd8209-cacerts\") pod \"istiod-openshift-gateway-7574459656-7fhcg\" (UID: \"27499250-8f4d-478c-945c-8b3512bd8209\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-7fhcg" Jul 19 07:48:56.077416 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:56.077048 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/27499250-8f4d-478c-945c-8b3512bd8209-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-7fhcg\" (UID: \"27499250-8f4d-478c-945c-8b3512bd8209\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-7fhcg" Jul 19 07:48:56.077416 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:56.077075 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6q4c\" (UniqueName: \"kubernetes.io/projected/27499250-8f4d-478c-945c-8b3512bd8209-kube-api-access-s6q4c\") pod \"istiod-openshift-gateway-7574459656-7fhcg\" (UID: \"27499250-8f4d-478c-945c-8b3512bd8209\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-7fhcg" Jul 19 07:48:56.077416 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:56.077122 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/27499250-8f4d-478c-945c-8b3512bd8209-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-7fhcg\" (UID: \"27499250-8f4d-478c-945c-8b3512bd8209\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-7fhcg" Jul 19 07:48:56.077604 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:56.077547 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/27499250-8f4d-478c-945c-8b3512bd8209-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-7fhcg\" (UID: \"27499250-8f4d-478c-945c-8b3512bd8209\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-7fhcg" Jul 19 07:48:56.081104 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:56.081074 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/27499250-8f4d-478c-945c-8b3512bd8209-cacerts\") pod \"istiod-openshift-gateway-7574459656-7fhcg\" (UID: \"27499250-8f4d-478c-945c-8b3512bd8209\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-7fhcg" Jul 19 07:48:56.081692 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:56.081652 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a402f5d0-4bcc-45be-9d79-13c7e577d3c3-tls-certs\") pod \"model-serving-api-54d89bd5fc-lvdn2\" (UID: \"a402f5d0-4bcc-45be-9d79-13c7e577d3c3\") " pod="opendatahub/model-serving-api-54d89bd5fc-lvdn2" Jul 19 07:48:56.081692 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:56.081674 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/27499250-8f4d-478c-945c-8b3512bd8209-local-certs\") pod \"istiod-openshift-gateway-7574459656-7fhcg\" (UID: \"27499250-8f4d-478c-945c-8b3512bd8209\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-7fhcg" Jul 19 07:48:56.081951 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:56.081920 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/27499250-8f4d-478c-945c-8b3512bd8209-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-7fhcg\" (UID: \"27499250-8f4d-478c-945c-8b3512bd8209\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-7fhcg" Jul 19 07:48:56.082209 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:56.082182 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/27499250-8f4d-478c-945c-8b3512bd8209-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-7fhcg\" (UID: \"27499250-8f4d-478c-945c-8b3512bd8209\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-7fhcg" Jul 19 07:48:56.085857 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:56.085809 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27499250-8f4d-478c-945c-8b3512bd8209-istio-token\") pod \"istiod-openshift-gateway-7574459656-7fhcg\" (UID: \"27499250-8f4d-478c-945c-8b3512bd8209\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-7fhcg" Jul 19 07:48:56.085966 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:56.085921 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm9fp\" (UniqueName: \"kubernetes.io/projected/a402f5d0-4bcc-45be-9d79-13c7e577d3c3-kube-api-access-dm9fp\") pod \"model-serving-api-54d89bd5fc-lvdn2\" (UID: \"a402f5d0-4bcc-45be-9d79-13c7e577d3c3\") " pod="opendatahub/model-serving-api-54d89bd5fc-lvdn2" Jul 19 07:48:56.086468 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:56.086447 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6q4c\" (UniqueName: \"kubernetes.io/projected/27499250-8f4d-478c-945c-8b3512bd8209-kube-api-access-s6q4c\") pod \"istiod-openshift-gateway-7574459656-7fhcg\" (UID: \"27499250-8f4d-478c-945c-8b3512bd8209\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-7fhcg" Jul 19 07:48:56.175591 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:56.175415 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-lvdn2" Jul 19 07:48:56.206010 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:56.205977 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-7fhcg" Jul 19 07:48:56.369438 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:56.367953 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-lvdn2"] Jul 19 07:48:56.371488 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:48:56.371451 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda402f5d0_4bcc_45be_9d79_13c7e577d3c3.slice/crio-8c5328eae77568e9bbf1adb670deebadd85e0f19049bba0b4803c60457bb2bac WatchSource:0}: Error finding container 8c5328eae77568e9bbf1adb670deebadd85e0f19049bba0b4803c60457bb2bac: Status 404 returned error can't find the container with id 8c5328eae77568e9bbf1adb670deebadd85e0f19049bba0b4803c60457bb2bac Jul 19 07:48:56.397611 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:56.397536 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-7fhcg"] Jul 19 07:48:56.400211 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:48:56.400175 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27499250_8f4d_478c_945c_8b3512bd8209.slice/crio-41e14d7aa4c938a737edcd789211227eb78be37c2181c901d621d136371a08fb WatchSource:0}: Error finding container 41e14d7aa4c938a737edcd789211227eb78be37c2181c901d621d136371a08fb: Status 404 returned error can't find the container with id 41e14d7aa4c938a737edcd789211227eb78be37c2181c901d621d136371a08fb Jul 19 07:48:56.585332 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:56.585223 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-56f59f876d-qqxn9" event={"ID":"1c5887e0-25ef-48da-a06c-a1874a64e2c7","Type":"ContainerStarted","Data":"50c3bc1e6a0ab87d36ba7c37961fb8aaacf793a114d11b4f8f4b4d22b17c43b3"} Jul 19 07:48:56.587182 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:56.587116 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-7fhcg" event={"ID":"27499250-8f4d-478c-945c-8b3512bd8209","Type":"ContainerStarted","Data":"41e14d7aa4c938a737edcd789211227eb78be37c2181c901d621d136371a08fb"} Jul 19 07:48:56.588844 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:56.588790 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-lvdn2" event={"ID":"a402f5d0-4bcc-45be-9d79-13c7e577d3c3","Type":"ContainerStarted","Data":"8c5328eae77568e9bbf1adb670deebadd85e0f19049bba0b4803c60457bb2bac"} Jul 19 07:48:56.746706 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:48:56.746648 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:49:00.611613 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:00.611567 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-85fwf" event={"ID":"d389a635-9a6d-42d5-b70c-b78a0898b929","Type":"ContainerStarted","Data":"d48f5589c0e09725168b65ef575d6ae5d4b31d76aa0e75e018ddfb914bd18fce"} Jul 19 07:49:00.612121 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:00.611674 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-85fwf" Jul 19 07:49:00.613391 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:00.613358 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-56f59f876d-qqxn9" event={"ID":"1c5887e0-25ef-48da-a06c-a1874a64e2c7","Type":"ContainerStarted","Data":"bbd709cdb163fe41e8f0a7a2d0b2f171c1ee97a127d40dfeae55aba446f2584f"} Jul 19 07:49:00.615081 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:00.615037 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-lvdn2" event={"ID":"a402f5d0-4bcc-45be-9d79-13c7e577d3c3","Type":"ContainerStarted","Data":"1749b4aa2e939dd368dee0a67cda61edd9e8d10b496858e7acc273d8ad65150f"} Jul 19 07:49:00.615234 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:00.615218 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-lvdn2" Jul 19 07:49:00.629770 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:00.629713 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-85fwf" podStartSLOduration=2.354543756 podStartE2EDuration="6.629693843s" totalImagesPullingTime="4.275150087s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 07:48:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:49:00.628086193 +0000 UTC m=+406.982395794" watchObservedRunningTime="2026-07-19 07:49:00.629693843 +0000 UTC m=+406.984003444" Jul 19 07:49:00.644462 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:00.644333 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-56f59f876d-qqxn9" podStartSLOduration=2.778960032 podStartE2EDuration="6.64431214s" totalImagesPullingTime="3.865352108s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 07:48:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:49:00.642171848 +0000 UTC m=+406.996481449" watchObservedRunningTime="2026-07-19 07:49:00.64431214 +0000 UTC m=+406.998621741" Jul 19 07:49:00.659759 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:00.659690 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-lvdn2" podStartSLOduration=2.339551321 podStartE2EDuration="5.659664426s" totalImagesPullingTime="3.320113105s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 07:48:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:49:00.658576565 +0000 UTC m=+407.012886169" watchObservedRunningTime="2026-07-19 07:49:00.659664426 +0000 UTC m=+407.013974029" Jul 19 07:49:01.103086 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:01.103048 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 19 07:49:01.103155 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:01.103119 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 19 07:49:01.622283 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:01.622238 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-7fhcg" event={"ID":"27499250-8f4d-478c-945c-8b3512bd8209","Type":"ContainerStarted","Data":"e232625b350a06c82ad8b99631262b521418f142e7d5e84d95eeb25fc400f91a"} Jul 19 07:49:01.622741 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:01.622544 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-7fhcg" Jul 19 07:49:01.644783 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:01.644724 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-7fhcg" podStartSLOduration=1.945224562 podStartE2EDuration="6.644706112s" totalImagesPullingTime="4.69948155s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 07:48:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:49:01.642922811 +0000 UTC m=+407.997232411" watchObservedRunningTime="2026-07-19 07:49:01.644706112 +0000 UTC m=+407.999015712" Jul 19 07:49:02.270692 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:02.270647 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:49:02.371233 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:02.371194 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:49:02.628780 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:02.628699 2574 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-7fhcg container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 19 07:49:02.628780 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:02.628752 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-7fhcg" podUID="27499250-8f4d-478c-945c-8b3512bd8209" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 19 07:49:03.078600 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:03.078560 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:49:03.265653 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:03.265610 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:49:03.373972 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:03.373876 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:49:05.627967 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:05.627937 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-7fhcg" Jul 19 07:49:06.106437 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:06.106396 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:49:06.239614 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:06.239576 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:49:06.260277 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:06.260240 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-gdc7l"] Jul 19 07:49:06.263830 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:06.263784 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-gdc7l" Jul 19 07:49:06.269596 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:06.269568 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 19 07:49:06.269750 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:06.269677 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-t46g4\"" Jul 19 07:49:06.287831 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:06.287801 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-gdc7l"] Jul 19 07:49:06.374681 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:06.374581 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/33b278bc-869f-4d53-9c01-78e3252c1fc8-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-gdc7l\" (UID: \"33b278bc-869f-4d53-9c01-78e3252c1fc8\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-gdc7l" Jul 19 07:49:06.374681 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:06.374622 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjhkh\" (UniqueName: \"kubernetes.io/projected/33b278bc-869f-4d53-9c01-78e3252c1fc8-kube-api-access-cjhkh\") pod \"llmisvc-controller-manager-777c6bd7d5-gdc7l\" (UID: \"33b278bc-869f-4d53-9c01-78e3252c1fc8\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-gdc7l" Jul 19 07:49:06.469855 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:06.469646 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:49:06.475562 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:06.475513 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/33b278bc-869f-4d53-9c01-78e3252c1fc8-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-gdc7l\" (UID: \"33b278bc-869f-4d53-9c01-78e3252c1fc8\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-gdc7l" Jul 19 07:49:06.475747 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:06.475571 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjhkh\" (UniqueName: \"kubernetes.io/projected/33b278bc-869f-4d53-9c01-78e3252c1fc8-kube-api-access-cjhkh\") pod \"llmisvc-controller-manager-777c6bd7d5-gdc7l\" (UID: \"33b278bc-869f-4d53-9c01-78e3252c1fc8\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-gdc7l" Jul 19 07:49:06.475747 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:49:06.475645 2574 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 19 07:49:06.475747 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:49:06.475724 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33b278bc-869f-4d53-9c01-78e3252c1fc8-cert podName:33b278bc-869f-4d53-9c01-78e3252c1fc8 nodeName:}" failed. No retries permitted until 2026-07-19 07:49:06.9757023 +0000 UTC m=+413.330011884 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/33b278bc-869f-4d53-9c01-78e3252c1fc8-cert") pod "llmisvc-controller-manager-777c6bd7d5-gdc7l" (UID: "33b278bc-869f-4d53-9c01-78e3252c1fc8") : secret "llmisvc-webhook-server-cert" not found Jul 19 07:49:06.489687 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:06.489652 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjhkh\" (UniqueName: \"kubernetes.io/projected/33b278bc-869f-4d53-9c01-78e3252c1fc8-kube-api-access-cjhkh\") pod \"llmisvc-controller-manager-777c6bd7d5-gdc7l\" (UID: \"33b278bc-869f-4d53-9c01-78e3252c1fc8\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-gdc7l" Jul 19 07:49:06.566272 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:06.566229 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:49:06.676740 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:06.676614 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:49:06.787159 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:06.787123 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:49:06.979813 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:06.979769 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/33b278bc-869f-4d53-9c01-78e3252c1fc8-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-gdc7l\" (UID: \"33b278bc-869f-4d53-9c01-78e3252c1fc8\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-gdc7l" Jul 19 07:49:06.982622 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:06.982578 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/33b278bc-869f-4d53-9c01-78e3252c1fc8-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-gdc7l\" (UID: \"33b278bc-869f-4d53-9c01-78e3252c1fc8\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-gdc7l" Jul 19 07:49:07.176233 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:07.176176 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-gdc7l" Jul 19 07:49:07.325316 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:07.325262 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-gdc7l"] Jul 19 07:49:07.329294 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:49:07.329257 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod33b278bc_869f_4d53_9c01_78e3252c1fc8.slice/crio-d37496b8f3885a5ee41b28e11b002f3c1a0cf495a093d2cdba27425e3d79ff90 WatchSource:0}: Error finding container d37496b8f3885a5ee41b28e11b002f3c1a0cf495a093d2cdba27425e3d79ff90: Status 404 returned error can't find the container with id d37496b8f3885a5ee41b28e11b002f3c1a0cf495a093d2cdba27425e3d79ff90 Jul 19 07:49:07.650956 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:07.650866 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-gdc7l" event={"ID":"33b278bc-869f-4d53-9c01-78e3252c1fc8","Type":"ContainerStarted","Data":"d37496b8f3885a5ee41b28e11b002f3c1a0cf495a093d2cdba27425e3d79ff90"} Jul 19 07:49:08.840721 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:08.840686 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:49:08.935156 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:08.935122 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:49:09.375194 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:09.375160 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-7fhcg"] Jul 19 07:49:09.375399 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:09.375372 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-7fhcg" podUID="27499250-8f4d-478c-945c-8b3512bd8209" containerName="discovery" containerID="cri-o://e232625b350a06c82ad8b99631262b521418f142e7d5e84d95eeb25fc400f91a" gracePeriod=30 Jul 19 07:49:09.910222 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:09.910196 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-7fhcg" Jul 19 07:49:10.017324 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:10.017236 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/27499250-8f4d-478c-945c-8b3512bd8209-istio-csr-dns-cert\") pod \"27499250-8f4d-478c-945c-8b3512bd8209\" (UID: \"27499250-8f4d-478c-945c-8b3512bd8209\") " Jul 19 07:49:10.017324 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:10.017288 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27499250-8f4d-478c-945c-8b3512bd8209-istio-token\") pod \"27499250-8f4d-478c-945c-8b3512bd8209\" (UID: \"27499250-8f4d-478c-945c-8b3512bd8209\") " Jul 19 07:49:10.017324 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:10.017325 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/27499250-8f4d-478c-945c-8b3512bd8209-cacerts\") pod \"27499250-8f4d-478c-945c-8b3512bd8209\" (UID: \"27499250-8f4d-478c-945c-8b3512bd8209\") " Jul 19 07:49:10.017608 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:10.017356 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/27499250-8f4d-478c-945c-8b3512bd8209-istio-kubeconfig\") pod \"27499250-8f4d-478c-945c-8b3512bd8209\" (UID: \"27499250-8f4d-478c-945c-8b3512bd8209\") " Jul 19 07:49:10.017608 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:10.017401 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/27499250-8f4d-478c-945c-8b3512bd8209-istio-csr-ca-configmap\") pod \"27499250-8f4d-478c-945c-8b3512bd8209\" (UID: \"27499250-8f4d-478c-945c-8b3512bd8209\") " Jul 19 07:49:10.017608 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:10.017439 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s6q4c\" (UniqueName: \"kubernetes.io/projected/27499250-8f4d-478c-945c-8b3512bd8209-kube-api-access-s6q4c\") pod \"27499250-8f4d-478c-945c-8b3512bd8209\" (UID: \"27499250-8f4d-478c-945c-8b3512bd8209\") " Jul 19 07:49:10.017608 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:10.017482 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/27499250-8f4d-478c-945c-8b3512bd8209-local-certs\") pod \"27499250-8f4d-478c-945c-8b3512bd8209\" (UID: \"27499250-8f4d-478c-945c-8b3512bd8209\") " Jul 19 07:49:10.017917 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:10.017875 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27499250-8f4d-478c-945c-8b3512bd8209-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "27499250-8f4d-478c-945c-8b3512bd8209" (UID: "27499250-8f4d-478c-945c-8b3512bd8209"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 07:49:10.020759 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:10.020728 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27499250-8f4d-478c-945c-8b3512bd8209-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "27499250-8f4d-478c-945c-8b3512bd8209" (UID: "27499250-8f4d-478c-945c-8b3512bd8209"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 07:49:10.020759 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:10.020748 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27499250-8f4d-478c-945c-8b3512bd8209-cacerts" (OuterVolumeSpecName: "cacerts") pod "27499250-8f4d-478c-945c-8b3512bd8209" (UID: "27499250-8f4d-478c-945c-8b3512bd8209"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 07:49:10.020909 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:10.020730 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27499250-8f4d-478c-945c-8b3512bd8209-local-certs" (OuterVolumeSpecName: "local-certs") pod "27499250-8f4d-478c-945c-8b3512bd8209" (UID: "27499250-8f4d-478c-945c-8b3512bd8209"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 07:49:10.020909 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:10.020730 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27499250-8f4d-478c-945c-8b3512bd8209-kube-api-access-s6q4c" (OuterVolumeSpecName: "kube-api-access-s6q4c") pod "27499250-8f4d-478c-945c-8b3512bd8209" (UID: "27499250-8f4d-478c-945c-8b3512bd8209"). InnerVolumeSpecName "kube-api-access-s6q4c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 07:49:10.020909 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:10.020791 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27499250-8f4d-478c-945c-8b3512bd8209-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "27499250-8f4d-478c-945c-8b3512bd8209" (UID: "27499250-8f4d-478c-945c-8b3512bd8209"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 07:49:10.020909 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:10.020832 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27499250-8f4d-478c-945c-8b3512bd8209-istio-token" (OuterVolumeSpecName: "istio-token") pod "27499250-8f4d-478c-945c-8b3512bd8209" (UID: "27499250-8f4d-478c-945c-8b3512bd8209"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 07:49:10.118610 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:10.118577 2574 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/27499250-8f4d-478c-945c-8b3512bd8209-istio-csr-ca-configmap\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 07:49:10.118610 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:10.118605 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6q4c\" (UniqueName: \"kubernetes.io/projected/27499250-8f4d-478c-945c-8b3512bd8209-kube-api-access-s6q4c\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 07:49:10.118610 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:10.118617 2574 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/27499250-8f4d-478c-945c-8b3512bd8209-local-certs\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 07:49:10.118850 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:10.118625 2574 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/27499250-8f4d-478c-945c-8b3512bd8209-istio-csr-dns-cert\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 07:49:10.118850 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:10.118634 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27499250-8f4d-478c-945c-8b3512bd8209-istio-token\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 07:49:10.118850 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:10.118642 2574 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/27499250-8f4d-478c-945c-8b3512bd8209-cacerts\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 07:49:10.118850 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:10.118649 2574 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/27499250-8f4d-478c-945c-8b3512bd8209-istio-kubeconfig\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 07:49:10.667959 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:10.667913 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-gdc7l" event={"ID":"33b278bc-869f-4d53-9c01-78e3252c1fc8","Type":"ContainerStarted","Data":"db2ffdf312a56fb5b6cbdef61082275cdadfc5fa2044e9aa7728324e317d2d0e"} Jul 19 07:49:10.668156 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:10.667993 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-gdc7l" Jul 19 07:49:10.669327 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:10.669302 2574 generic.go:356] "Generic (PLEG): container finished" podID="27499250-8f4d-478c-945c-8b3512bd8209" containerID="e232625b350a06c82ad8b99631262b521418f142e7d5e84d95eeb25fc400f91a" exitCode=0 Jul 19 07:49:10.669434 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:10.669354 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-7fhcg" Jul 19 07:49:10.669434 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:10.669356 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-7fhcg" event={"ID":"27499250-8f4d-478c-945c-8b3512bd8209","Type":"ContainerDied","Data":"e232625b350a06c82ad8b99631262b521418f142e7d5e84d95eeb25fc400f91a"} Jul 19 07:49:10.669506 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:10.669461 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-7fhcg" event={"ID":"27499250-8f4d-478c-945c-8b3512bd8209","Type":"ContainerDied","Data":"41e14d7aa4c938a737edcd789211227eb78be37c2181c901d621d136371a08fb"} Jul 19 07:49:10.669506 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:10.669485 2574 scope.go:117] "RemoveContainer" containerID="e232625b350a06c82ad8b99631262b521418f142e7d5e84d95eeb25fc400f91a" Jul 19 07:49:10.678207 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:10.678188 2574 scope.go:117] "RemoveContainer" containerID="e232625b350a06c82ad8b99631262b521418f142e7d5e84d95eeb25fc400f91a" Jul 19 07:49:10.678449 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:49:10.678430 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e232625b350a06c82ad8b99631262b521418f142e7d5e84d95eeb25fc400f91a\": container with ID starting with e232625b350a06c82ad8b99631262b521418f142e7d5e84d95eeb25fc400f91a not found: ID does not exist" containerID="e232625b350a06c82ad8b99631262b521418f142e7d5e84d95eeb25fc400f91a" Jul 19 07:49:10.678493 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:10.678455 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e232625b350a06c82ad8b99631262b521418f142e7d5e84d95eeb25fc400f91a"} err="failed to get container status \"e232625b350a06c82ad8b99631262b521418f142e7d5e84d95eeb25fc400f91a\": rpc error: code = NotFound desc = could not find container \"e232625b350a06c82ad8b99631262b521418f142e7d5e84d95eeb25fc400f91a\": container with ID starting with e232625b350a06c82ad8b99631262b521418f142e7d5e84d95eeb25fc400f91a not found: ID does not exist" Jul 19 07:49:10.698047 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:10.698001 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-gdc7l" podStartSLOduration=2.210632801 podStartE2EDuration="4.697987452s" totalImagesPullingTime="2.487354651s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 07:49:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:49:10.696200674 +0000 UTC m=+417.050510275" watchObservedRunningTime="2026-07-19 07:49:10.697987452 +0000 UTC m=+417.052297053" Jul 19 07:49:10.711336 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:10.711308 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-7fhcg"] Jul 19 07:49:10.715140 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:10.715115 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-7fhcg"] Jul 19 07:49:11.625537 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:11.625488 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-85fwf" Jul 19 07:49:11.627723 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:11.627694 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-lvdn2" Jul 19 07:49:12.558699 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:12.558657 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27499250-8f4d-478c-945c-8b3512bd8209" path="/var/lib/kubelet/pods/27499250-8f4d-478c-945c-8b3512bd8209/volumes" Jul 19 07:49:17.555504 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:17.555454 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:49:17.610045 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:17.610009 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:49:27.489872 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:27.489827 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:49:29.338125 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:29.338081 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:49:29.397158 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:29.397121 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:49:41.678567 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:41.678461 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-gdc7l" Jul 19 07:49:55.497146 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:55.497071 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:49:55.603854 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:55.603816 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:49:55.622895 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:55.622856 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-4fc6p"] Jul 19 07:49:55.623198 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:55.623185 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27499250-8f4d-478c-945c-8b3512bd8209" containerName="discovery" Jul 19 07:49:55.623246 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:55.623199 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="27499250-8f4d-478c-945c-8b3512bd8209" containerName="discovery" Jul 19 07:49:55.623280 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:55.623258 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="27499250-8f4d-478c-945c-8b3512bd8209" containerName="discovery" Jul 19 07:49:55.630823 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:55.630798 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-4fc6p" Jul 19 07:49:55.633575 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:55.633542 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 19 07:49:55.633742 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:55.633514 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-44wm8\"" Jul 19 07:49:55.634242 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:55.634219 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-4fc6p"] Jul 19 07:49:55.634469 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:55.634435 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 19 07:49:55.668566 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:55.668512 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:49:55.812726 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:55.812629 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdjns\" (UniqueName: \"kubernetes.io/projected/ed672a56-5088-4a24-9675-a5706dcc723c-kube-api-access-vdjns\") pod \"kuadrant-operator-catalog-4fc6p\" (UID: \"ed672a56-5088-4a24-9675-a5706dcc723c\") " pod="kuadrant-system/kuadrant-operator-catalog-4fc6p" Jul 19 07:49:55.913584 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:55.913551 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdjns\" (UniqueName: \"kubernetes.io/projected/ed672a56-5088-4a24-9675-a5706dcc723c-kube-api-access-vdjns\") pod \"kuadrant-operator-catalog-4fc6p\" (UID: \"ed672a56-5088-4a24-9675-a5706dcc723c\") " pod="kuadrant-system/kuadrant-operator-catalog-4fc6p" Jul 19 07:49:55.921185 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:55.921152 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdjns\" (UniqueName: \"kubernetes.io/projected/ed672a56-5088-4a24-9675-a5706dcc723c-kube-api-access-vdjns\") pod \"kuadrant-operator-catalog-4fc6p\" (UID: \"ed672a56-5088-4a24-9675-a5706dcc723c\") " pod="kuadrant-system/kuadrant-operator-catalog-4fc6p" Jul 19 07:49:55.942217 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:55.942174 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-4fc6p" Jul 19 07:49:55.988892 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:55.988855 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-4fc6p"] Jul 19 07:49:56.075786 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:56.075761 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-4fc6p"] Jul 19 07:49:56.078627 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:49:56.078598 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded672a56_5088_4a24_9675_a5706dcc723c.slice/crio-04bbbd13a3fecc7c60680d9db240219067308d8e8350382f4fc982a0427f06ca WatchSource:0}: Error finding container 04bbbd13a3fecc7c60680d9db240219067308d8e8350382f4fc982a0427f06ca: Status 404 returned error can't find the container with id 04bbbd13a3fecc7c60680d9db240219067308d8e8350382f4fc982a0427f06ca Jul 19 07:49:56.197853 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:56.197817 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-ttzxt"] Jul 19 07:49:56.203041 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:56.203021 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-ttzxt" Jul 19 07:49:56.208920 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:56.208891 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-ttzxt"] Jul 19 07:49:56.317308 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:56.317271 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8ptd\" (UniqueName: \"kubernetes.io/projected/2b0ebf15-5daf-47dd-8acb-3a668927d267-kube-api-access-l8ptd\") pod \"kuadrant-operator-catalog-ttzxt\" (UID: \"2b0ebf15-5daf-47dd-8acb-3a668927d267\") " pod="kuadrant-system/kuadrant-operator-catalog-ttzxt" Jul 19 07:49:56.417793 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:56.417704 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8ptd\" (UniqueName: \"kubernetes.io/projected/2b0ebf15-5daf-47dd-8acb-3a668927d267-kube-api-access-l8ptd\") pod \"kuadrant-operator-catalog-ttzxt\" (UID: \"2b0ebf15-5daf-47dd-8acb-3a668927d267\") " pod="kuadrant-system/kuadrant-operator-catalog-ttzxt" Jul 19 07:49:56.420699 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:56.420668 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:49:56.425993 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:56.425963 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8ptd\" (UniqueName: \"kubernetes.io/projected/2b0ebf15-5daf-47dd-8acb-3a668927d267-kube-api-access-l8ptd\") pod \"kuadrant-operator-catalog-ttzxt\" (UID: \"2b0ebf15-5daf-47dd-8acb-3a668927d267\") " pod="kuadrant-system/kuadrant-operator-catalog-ttzxt" Jul 19 07:49:56.514240 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:56.514203 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-ttzxt" Jul 19 07:49:56.619427 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:56.619389 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:49:56.654708 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:56.654599 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-ttzxt"] Jul 19 07:49:56.657063 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:49:56.657032 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b0ebf15_5daf_47dd_8acb_3a668927d267.slice/crio-c26df979e172d34ebe8d46577be4a7753dacb2a93b671b214f9068f095992ab9 WatchSource:0}: Error finding container c26df979e172d34ebe8d46577be4a7753dacb2a93b671b214f9068f095992ab9: Status 404 returned error can't find the container with id c26df979e172d34ebe8d46577be4a7753dacb2a93b671b214f9068f095992ab9 Jul 19 07:49:56.882859 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:56.882822 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-ttzxt" event={"ID":"2b0ebf15-5daf-47dd-8acb-3a668927d267","Type":"ContainerStarted","Data":"c26df979e172d34ebe8d46577be4a7753dacb2a93b671b214f9068f095992ab9"} Jul 19 07:49:56.883877 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:56.883854 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-4fc6p" event={"ID":"ed672a56-5088-4a24-9675-a5706dcc723c","Type":"ContainerStarted","Data":"04bbbd13a3fecc7c60680d9db240219067308d8e8350382f4fc982a0427f06ca"} Jul 19 07:49:58.894536 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:58.894482 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-ttzxt" event={"ID":"2b0ebf15-5daf-47dd-8acb-3a668927d267","Type":"ContainerStarted","Data":"c17895ac6ec829359f6b2a8ba138b7b1ca736638b7379b2afb9662eca5eeca5f"} Jul 19 07:49:58.895936 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:58.895908 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-4fc6p" event={"ID":"ed672a56-5088-4a24-9675-a5706dcc723c","Type":"ContainerStarted","Data":"1f4dc378206d07f097c9310a466ec32220321f7f123b03db78f44ab5ccf489ca"} Jul 19 07:49:58.896051 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:58.895981 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-4fc6p" podUID="ed672a56-5088-4a24-9675-a5706dcc723c" containerName="registry-server" containerID="cri-o://1f4dc378206d07f097c9310a466ec32220321f7f123b03db78f44ab5ccf489ca" gracePeriod=2 Jul 19 07:49:58.909328 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:58.909286 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-ttzxt" podStartSLOduration=1.366326945 podStartE2EDuration="2.909273251s" totalImagesPullingTime="1.542946306s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 07:49:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:49:58.90782498 +0000 UTC m=+465.262134581" watchObservedRunningTime="2026-07-19 07:49:58.909273251 +0000 UTC m=+465.263582858" Jul 19 07:49:58.922093 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:58.922051 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-4fc6p" podStartSLOduration=1.8013711140000002 podStartE2EDuration="3.922035918s" totalImagesPullingTime="2.120664804s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 07:49:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:49:58.921293413 +0000 UTC m=+465.275603014" watchObservedRunningTime="2026-07-19 07:49:58.922035918 +0000 UTC m=+465.276345519" Jul 19 07:49:59.141842 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:59.141816 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-4fc6p" Jul 19 07:49:59.241822 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:59.241790 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vdjns\" (UniqueName: \"kubernetes.io/projected/ed672a56-5088-4a24-9675-a5706dcc723c-kube-api-access-vdjns\") pod \"ed672a56-5088-4a24-9675-a5706dcc723c\" (UID: \"ed672a56-5088-4a24-9675-a5706dcc723c\") " Jul 19 07:49:59.244093 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:59.244057 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed672a56-5088-4a24-9675-a5706dcc723c-kube-api-access-vdjns" (OuterVolumeSpecName: "kube-api-access-vdjns") pod "ed672a56-5088-4a24-9675-a5706dcc723c" (UID: "ed672a56-5088-4a24-9675-a5706dcc723c"). InnerVolumeSpecName "kube-api-access-vdjns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 07:49:59.342672 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:59.342630 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdjns\" (UniqueName: \"kubernetes.io/projected/ed672a56-5088-4a24-9675-a5706dcc723c-kube-api-access-vdjns\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 07:49:59.901760 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:59.901724 2574 generic.go:356] "Generic (PLEG): container finished" podID="ed672a56-5088-4a24-9675-a5706dcc723c" containerID="1f4dc378206d07f097c9310a466ec32220321f7f123b03db78f44ab5ccf489ca" exitCode=0 Jul 19 07:49:59.902172 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:59.901769 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-4fc6p" Jul 19 07:49:59.902172 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:59.901809 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-4fc6p" event={"ID":"ed672a56-5088-4a24-9675-a5706dcc723c","Type":"ContainerDied","Data":"1f4dc378206d07f097c9310a466ec32220321f7f123b03db78f44ab5ccf489ca"} Jul 19 07:49:59.902172 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:59.901860 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-4fc6p" event={"ID":"ed672a56-5088-4a24-9675-a5706dcc723c","Type":"ContainerDied","Data":"04bbbd13a3fecc7c60680d9db240219067308d8e8350382f4fc982a0427f06ca"} Jul 19 07:49:59.902172 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:59.901885 2574 scope.go:117] "RemoveContainer" containerID="1f4dc378206d07f097c9310a466ec32220321f7f123b03db78f44ab5ccf489ca" Jul 19 07:49:59.911467 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:59.911447 2574 scope.go:117] "RemoveContainer" containerID="1f4dc378206d07f097c9310a466ec32220321f7f123b03db78f44ab5ccf489ca" Jul 19 07:49:59.911771 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:49:59.911750 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f4dc378206d07f097c9310a466ec32220321f7f123b03db78f44ab5ccf489ca\": container with ID starting with 1f4dc378206d07f097c9310a466ec32220321f7f123b03db78f44ab5ccf489ca not found: ID does not exist" containerID="1f4dc378206d07f097c9310a466ec32220321f7f123b03db78f44ab5ccf489ca" Jul 19 07:49:59.911841 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:59.911781 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f4dc378206d07f097c9310a466ec32220321f7f123b03db78f44ab5ccf489ca"} err="failed to get container status \"1f4dc378206d07f097c9310a466ec32220321f7f123b03db78f44ab5ccf489ca\": rpc error: code = NotFound desc = could not find container \"1f4dc378206d07f097c9310a466ec32220321f7f123b03db78f44ab5ccf489ca\": container with ID starting with 1f4dc378206d07f097c9310a466ec32220321f7f123b03db78f44ab5ccf489ca not found: ID does not exist" Jul 19 07:49:59.924231 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:59.924178 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-4fc6p"] Jul 19 07:49:59.925773 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:49:59.925752 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-4fc6p"] Jul 19 07:50:00.558063 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:00.558029 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed672a56-5088-4a24-9675-a5706dcc723c" path="/var/lib/kubelet/pods/ed672a56-5088-4a24-9675-a5706dcc723c/volumes" Jul 19 07:50:06.515262 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:06.515219 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-ttzxt" Jul 19 07:50:06.515706 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:06.515319 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-ttzxt" Jul 19 07:50:06.538161 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:06.538129 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-ttzxt" Jul 19 07:50:06.956436 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:06.956346 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-ttzxt" Jul 19 07:50:24.416488 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:24.416450 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:50:24.521109 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:24.521044 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:50:24.616392 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:24.616359 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:50:25.049285 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:25.049250 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:50:25.157859 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:25.157819 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:50:25.261234 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:25.261194 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:50:27.664045 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:27.664012 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-ghjrd"] Jul 19 07:50:27.664445 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:27.664359 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed672a56-5088-4a24-9675-a5706dcc723c" containerName="registry-server" Jul 19 07:50:27.664445 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:27.664369 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed672a56-5088-4a24-9675-a5706dcc723c" containerName="registry-server" Jul 19 07:50:27.664445 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:27.664431 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed672a56-5088-4a24-9675-a5706dcc723c" containerName="registry-server" Jul 19 07:50:27.668772 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:27.668755 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-ghjrd" Jul 19 07:50:27.671512 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:27.671492 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 19 07:50:27.671613 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:27.671555 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-wg6vc\"" Jul 19 07:50:27.676760 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:27.676737 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-ghjrd"] Jul 19 07:50:27.788547 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:27.788487 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qp9cj\" (UniqueName: \"kubernetes.io/projected/727df024-63b4-4a1e-81a6-b546c6ebda2d-kube-api-access-qp9cj\") pod \"dns-operator-controller-manager-864dd78c4c-ghjrd\" (UID: \"727df024-63b4-4a1e-81a6-b546c6ebda2d\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-ghjrd" Jul 19 07:50:27.889057 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:27.889022 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qp9cj\" (UniqueName: \"kubernetes.io/projected/727df024-63b4-4a1e-81a6-b546c6ebda2d-kube-api-access-qp9cj\") pod \"dns-operator-controller-manager-864dd78c4c-ghjrd\" (UID: \"727df024-63b4-4a1e-81a6-b546c6ebda2d\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-ghjrd" Jul 19 07:50:27.903337 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:27.903305 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qp9cj\" (UniqueName: \"kubernetes.io/projected/727df024-63b4-4a1e-81a6-b546c6ebda2d-kube-api-access-qp9cj\") pod \"dns-operator-controller-manager-864dd78c4c-ghjrd\" (UID: \"727df024-63b4-4a1e-81a6-b546c6ebda2d\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-ghjrd" Jul 19 07:50:27.979425 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:27.979369 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-ghjrd" Jul 19 07:50:28.115043 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:28.115019 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-ghjrd"] Jul 19 07:50:28.117470 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:50:28.117439 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod727df024_63b4_4a1e_81a6_b546c6ebda2d.slice/crio-10e4cb40ff1c0ab4e9e5c5bd7167fa7b58233f988ec0f1a003e26c5e123d7d52 WatchSource:0}: Error finding container 10e4cb40ff1c0ab4e9e5c5bd7167fa7b58233f988ec0f1a003e26c5e123d7d52: Status 404 returned error can't find the container with id 10e4cb40ff1c0ab4e9e5c5bd7167fa7b58233f988ec0f1a003e26c5e123d7d52 Jul 19 07:50:28.633076 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:28.633033 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:50:29.038118 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:29.038071 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-ghjrd" event={"ID":"727df024-63b4-4a1e-81a6-b546c6ebda2d","Type":"ContainerStarted","Data":"10e4cb40ff1c0ab4e9e5c5bd7167fa7b58233f988ec0f1a003e26c5e123d7d52"} Jul 19 07:50:30.044434 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:30.044390 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-ghjrd" event={"ID":"727df024-63b4-4a1e-81a6-b546c6ebda2d","Type":"ContainerStarted","Data":"0420dac556e5f0afc8e56e619a2c120e079e63f41feda6bd531cc15ea1ee77f4"} Jul 19 07:50:30.044863 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:30.044500 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-ghjrd" Jul 19 07:50:30.073922 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:30.073813 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-ghjrd" podStartSLOduration=1.369196133 podStartE2EDuration="3.073795282s" totalImagesPullingTime="1.704599149s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 07:50:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:50:30.071699115 +0000 UTC m=+496.426008716" watchObservedRunningTime="2026-07-19 07:50:30.073795282 +0000 UTC m=+496.428104882" Jul 19 07:50:31.662109 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:31.662074 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:50:32.672086 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:32.672051 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qc5mf"] Jul 19 07:50:32.680008 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:32.679983 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qc5mf" Jul 19 07:50:32.682752 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:32.682731 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-zktnv\"" Jul 19 07:50:32.690120 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:32.690095 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qc5mf"] Jul 19 07:50:32.731288 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:32.731250 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3dd100ae-9e01-4083-af93-87b3acc4d914-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-qc5mf\" (UID: \"3dd100ae-9e01-4083-af93-87b3acc4d914\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qc5mf" Jul 19 07:50:32.731477 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:32.731329 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsstq\" (UniqueName: \"kubernetes.io/projected/3dd100ae-9e01-4083-af93-87b3acc4d914-kube-api-access-rsstq\") pod \"kuadrant-operator-controller-manager-8684c9ff49-qc5mf\" (UID: \"3dd100ae-9e01-4083-af93-87b3acc4d914\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qc5mf" Jul 19 07:50:32.832430 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:32.832385 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsstq\" (UniqueName: \"kubernetes.io/projected/3dd100ae-9e01-4083-af93-87b3acc4d914-kube-api-access-rsstq\") pod \"kuadrant-operator-controller-manager-8684c9ff49-qc5mf\" (UID: \"3dd100ae-9e01-4083-af93-87b3acc4d914\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qc5mf" Jul 19 07:50:32.832627 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:32.832460 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3dd100ae-9e01-4083-af93-87b3acc4d914-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-qc5mf\" (UID: \"3dd100ae-9e01-4083-af93-87b3acc4d914\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qc5mf" Jul 19 07:50:32.832872 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:32.832852 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3dd100ae-9e01-4083-af93-87b3acc4d914-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-qc5mf\" (UID: \"3dd100ae-9e01-4083-af93-87b3acc4d914\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qc5mf" Jul 19 07:50:32.843773 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:32.843749 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsstq\" (UniqueName: \"kubernetes.io/projected/3dd100ae-9e01-4083-af93-87b3acc4d914-kube-api-access-rsstq\") pod \"kuadrant-operator-controller-manager-8684c9ff49-qc5mf\" (UID: \"3dd100ae-9e01-4083-af93-87b3acc4d914\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qc5mf" Jul 19 07:50:32.990504 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:32.990464 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qc5mf" Jul 19 07:50:33.097808 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:33.097753 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:50:33.136062 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:33.136029 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qc5mf"] Jul 19 07:50:33.139606 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:50:33.139579 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dd100ae_9e01_4083_af93_87b3acc4d914.slice/crio-7e8de0082c35453028b3b64cf6d954a6bf6fada7d62c5926114e0913b8744710 WatchSource:0}: Error finding container 7e8de0082c35453028b3b64cf6d954a6bf6fada7d62c5926114e0913b8744710: Status 404 returned error can't find the container with id 7e8de0082c35453028b3b64cf6d954a6bf6fada7d62c5926114e0913b8744710 Jul 19 07:50:33.239100 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:33.239061 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:50:33.829905 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:33.829866 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:50:34.065124 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:34.065083 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qc5mf" event={"ID":"3dd100ae-9e01-4083-af93-87b3acc4d914","Type":"ContainerStarted","Data":"7e8de0082c35453028b3b64cf6d954a6bf6fada7d62c5926114e0913b8744710"} Jul 19 07:50:34.871627 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:34.871587 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:50:34.969641 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:34.969606 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:50:35.075540 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:35.075351 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:50:36.432940 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:36.432900 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-7f54b"] Jul 19 07:50:36.436919 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:36.436893 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-7f54b" Jul 19 07:50:36.439894 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:36.439860 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-5k7dm\"" Jul 19 07:50:36.447636 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:36.447607 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-7f54b"] Jul 19 07:50:36.480414 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:36.480342 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:50:36.564432 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:36.564384 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8qm6\" (UniqueName: \"kubernetes.io/projected/13322af6-4caf-4e03-89fe-4491af5d4255-kube-api-access-n8qm6\") pod \"authorino-operator-75cc8d5cb7-7f54b\" (UID: \"13322af6-4caf-4e03-89fe-4491af5d4255\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-7f54b" Jul 19 07:50:36.665513 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:36.665432 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8qm6\" (UniqueName: \"kubernetes.io/projected/13322af6-4caf-4e03-89fe-4491af5d4255-kube-api-access-n8qm6\") pod \"authorino-operator-75cc8d5cb7-7f54b\" (UID: \"13322af6-4caf-4e03-89fe-4491af5d4255\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-7f54b" Jul 19 07:50:36.681186 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:36.681151 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8qm6\" (UniqueName: \"kubernetes.io/projected/13322af6-4caf-4e03-89fe-4491af5d4255-kube-api-access-n8qm6\") pod \"authorino-operator-75cc8d5cb7-7f54b\" (UID: \"13322af6-4caf-4e03-89fe-4491af5d4255\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-7f54b" Jul 19 07:50:36.751322 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:36.751285 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-7f54b" Jul 19 07:50:36.893664 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:36.893632 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-7f54b"] Jul 19 07:50:36.895572 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:50:36.895536 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13322af6_4caf_4e03_89fe_4491af5d4255.slice/crio-85b982c8550a84d6f7b20eeadb2c71f11614dde61b732eda46e5c8ce84f7bb14 WatchSource:0}: Error finding container 85b982c8550a84d6f7b20eeadb2c71f11614dde61b732eda46e5c8ce84f7bb14: Status 404 returned error can't find the container with id 85b982c8550a84d6f7b20eeadb2c71f11614dde61b732eda46e5c8ce84f7bb14 Jul 19 07:50:37.080733 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:37.080645 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-7f54b" event={"ID":"13322af6-4caf-4e03-89fe-4491af5d4255","Type":"ContainerStarted","Data":"85b982c8550a84d6f7b20eeadb2c71f11614dde61b732eda46e5c8ce84f7bb14"} Jul 19 07:50:37.444654 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:37.444513 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:50:39.258029 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:39.257966 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lkx5c"] Jul 19 07:50:39.267989 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:39.267961 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lkx5c" Jul 19 07:50:39.270643 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:39.270614 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-t8whz\"" Jul 19 07:50:39.271273 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:39.271242 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lkx5c"] Jul 19 07:50:39.290147 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:39.290115 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:50:39.391224 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:39.391183 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpmvj\" (UniqueName: \"kubernetes.io/projected/a86703d2-7ae6-4c8d-ab15-6a8fbfffef48-kube-api-access-gpmvj\") pod \"limitador-operator-controller-manager-859d7b78bd-lkx5c\" (UID: \"a86703d2-7ae6-4c8d-ab15-6a8fbfffef48\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lkx5c" Jul 19 07:50:39.491757 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:39.491716 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpmvj\" (UniqueName: \"kubernetes.io/projected/a86703d2-7ae6-4c8d-ab15-6a8fbfffef48-kube-api-access-gpmvj\") pod \"limitador-operator-controller-manager-859d7b78bd-lkx5c\" (UID: \"a86703d2-7ae6-4c8d-ab15-6a8fbfffef48\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lkx5c" Jul 19 07:50:39.500785 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:39.500751 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpmvj\" (UniqueName: \"kubernetes.io/projected/a86703d2-7ae6-4c8d-ab15-6a8fbfffef48-kube-api-access-gpmvj\") pod \"limitador-operator-controller-manager-859d7b78bd-lkx5c\" (UID: \"a86703d2-7ae6-4c8d-ab15-6a8fbfffef48\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lkx5c" Jul 19 07:50:39.582963 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:39.582884 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lkx5c" Jul 19 07:50:40.010146 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:40.010117 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lkx5c"] Jul 19 07:50:40.015628 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:50:40.015599 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda86703d2_7ae6_4c8d_ab15_6a8fbfffef48.slice/crio-c852e8620c295ca08f2a30feeb92f2441f0c417a1fb7693ca7db4d3168863426 WatchSource:0}: Error finding container c852e8620c295ca08f2a30feeb92f2441f0c417a1fb7693ca7db4d3168863426: Status 404 returned error can't find the container with id c852e8620c295ca08f2a30feeb92f2441f0c417a1fb7693ca7db4d3168863426 Jul 19 07:50:40.099082 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:40.099044 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-7f54b" event={"ID":"13322af6-4caf-4e03-89fe-4491af5d4255","Type":"ContainerStarted","Data":"0b98e5f668ebc390e7041925602816218555cce1874282f522b86cb65b1a2ecb"} Jul 19 07:50:40.099295 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:40.099163 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-7f54b" Jul 19 07:50:40.100802 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:40.100778 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qc5mf" event={"ID":"3dd100ae-9e01-4083-af93-87b3acc4d914","Type":"ContainerStarted","Data":"bb0708b6a1a83ec6bce9d2fa144c0c10ebad096c918247f006fe0bb18d751033"} Jul 19 07:50:40.100891 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:40.100821 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qc5mf" Jul 19 07:50:40.102213 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:40.102190 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lkx5c" event={"ID":"a86703d2-7ae6-4c8d-ab15-6a8fbfffef48","Type":"ContainerStarted","Data":"c852e8620c295ca08f2a30feeb92f2441f0c417a1fb7693ca7db4d3168863426"} Jul 19 07:50:40.116682 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:40.116642 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-7f54b" podStartSLOduration=1.093648497 podStartE2EDuration="4.116626936s" totalImagesPullingTime="3.022978439s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 07:50:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:50:40.115076602 +0000 UTC m=+506.469386204" watchObservedRunningTime="2026-07-19 07:50:40.116626936 +0000 UTC m=+506.470936536" Jul 19 07:50:40.137751 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:40.133333 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qc5mf" podStartSLOduration=1.364498678 podStartE2EDuration="8.133312413s" totalImagesPullingTime="6.768813735s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 07:50:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:50:40.131644436 +0000 UTC m=+506.485954037" watchObservedRunningTime="2026-07-19 07:50:40.133312413 +0000 UTC m=+506.487622018" Jul 19 07:50:40.843414 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:40.843344 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:50:40.940420 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:40.940380 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:50:41.051670 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:41.051639 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-ghjrd" Jul 19 07:50:42.114153 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:42.114107 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lkx5c" event={"ID":"a86703d2-7ae6-4c8d-ab15-6a8fbfffef48","Type":"ContainerStarted","Data":"521162a6d716530351a4acc02c45f98bb53c2eb320d5e8501ba2d21531cbf365"} Jul 19 07:50:42.114591 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:42.114190 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lkx5c" Jul 19 07:50:42.132956 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:42.132894 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lkx5c" podStartSLOduration=1.6075692209999999 podStartE2EDuration="3.132874976s" totalImagesPullingTime="1.525305755s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 07:50:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:50:42.129995752 +0000 UTC m=+508.484305355" watchObservedRunningTime="2026-07-19 07:50:42.132874976 +0000 UTC m=+508.487184579" Jul 19 07:50:42.858846 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:42.858808 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:50:43.213438 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:43.213389 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:50:44.427854 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:44.427816 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:50:44.488782 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:44.488746 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:50:45.742250 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:45.742212 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:50:51.110698 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:51.110667 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qc5mf" Jul 19 07:50:51.111161 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:51.110747 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-7f54b" Jul 19 07:50:52.478572 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:52.478485 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-4nvz2"] Jul 19 07:50:52.483856 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:52.483820 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-4nvz2" Jul 19 07:50:52.521834 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:52.521801 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-4nvz2"] Jul 19 07:50:52.603565 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:52.603494 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/823547f7-9ebd-43d8-ad54-172d8c8c29f2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-4nvz2\" (UID: \"823547f7-9ebd-43d8-ad54-172d8c8c29f2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-4nvz2" Jul 19 07:50:52.603764 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:52.603611 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l28jv\" (UniqueName: \"kubernetes.io/projected/823547f7-9ebd-43d8-ad54-172d8c8c29f2-kube-api-access-l28jv\") pod \"kuadrant-operator-controller-manager-55467d9f7-4nvz2\" (UID: \"823547f7-9ebd-43d8-ad54-172d8c8c29f2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-4nvz2" Jul 19 07:50:52.704821 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:52.704783 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l28jv\" (UniqueName: \"kubernetes.io/projected/823547f7-9ebd-43d8-ad54-172d8c8c29f2-kube-api-access-l28jv\") pod \"kuadrant-operator-controller-manager-55467d9f7-4nvz2\" (UID: \"823547f7-9ebd-43d8-ad54-172d8c8c29f2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-4nvz2" Jul 19 07:50:52.705027 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:52.704928 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/823547f7-9ebd-43d8-ad54-172d8c8c29f2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-4nvz2\" (UID: \"823547f7-9ebd-43d8-ad54-172d8c8c29f2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-4nvz2" Jul 19 07:50:52.705378 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:52.705357 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/823547f7-9ebd-43d8-ad54-172d8c8c29f2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-4nvz2\" (UID: \"823547f7-9ebd-43d8-ad54-172d8c8c29f2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-4nvz2" Jul 19 07:50:52.713917 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:52.713890 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l28jv\" (UniqueName: \"kubernetes.io/projected/823547f7-9ebd-43d8-ad54-172d8c8c29f2-kube-api-access-l28jv\") pod \"kuadrant-operator-controller-manager-55467d9f7-4nvz2\" (UID: \"823547f7-9ebd-43d8-ad54-172d8c8c29f2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-4nvz2" Jul 19 07:50:52.793705 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:52.793608 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-4nvz2" Jul 19 07:50:52.863375 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:52.863329 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-4nvz2"] Jul 19 07:50:52.869753 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:52.869721 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-4nvz2"] Jul 19 07:50:52.887205 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:52.887168 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-t7dxv"] Jul 19 07:50:52.892203 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:52.892182 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-t7dxv" Jul 19 07:50:52.903137 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:52.903083 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-t7dxv"] Jul 19 07:50:52.906292 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:52.906266 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qc5mf"] Jul 19 07:50:52.907470 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:52.906480 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qc5mf" podUID="3dd100ae-9e01-4083-af93-87b3acc4d914" containerName="manager" containerID="cri-o://bb0708b6a1a83ec6bce9d2fa144c0c10ebad096c918247f006fe0bb18d751033" gracePeriod=2 Jul 19 07:50:52.919659 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:50:52.919622 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-qc5mf\" is forbidden: User \"system:node:ip-10-0-136-70.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-70.ec2.internal' and this object" podUID="3dd100ae-9e01-4083-af93-87b3acc4d914" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qc5mf" Jul 19 07:50:52.923226 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:52.923197 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qc5mf"] Jul 19 07:50:52.931461 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:52.931432 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lkx5c"] Jul 19 07:50:52.931685 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:52.931663 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lkx5c" podUID="a86703d2-7ae6-4c8d-ab15-6a8fbfffef48" containerName="manager" containerID="cri-o://521162a6d716530351a4acc02c45f98bb53c2eb320d5e8501ba2d21531cbf365" gracePeriod=2 Jul 19 07:50:52.933854 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:52.933834 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lkx5c" Jul 19 07:50:52.937376 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:52.937354 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lkx5c"] Jul 19 07:50:52.939752 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:50:52.939721 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-qc5mf\" is forbidden: User \"system:node:ip-10-0-136-70.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-70.ec2.internal' and this object" podUID="3dd100ae-9e01-4083-af93-87b3acc4d914" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qc5mf" Jul 19 07:50:52.941994 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:50:52.941966 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-qc5mf\" is forbidden: User \"system:node:ip-10-0-136-70.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-70.ec2.internal' and this object" podUID="3dd100ae-9e01-4083-af93-87b3acc4d914" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qc5mf" Jul 19 07:50:52.944010 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:50:52.943975 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-lkx5c\" is forbidden: User \"system:node:ip-10-0-136-70.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-70.ec2.internal' and this object" podUID="a86703d2-7ae6-4c8d-ab15-6a8fbfffef48" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lkx5c" Jul 19 07:50:52.956129 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:52.956102 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ssftg"] Jul 19 07:50:52.956506 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:52.956491 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a86703d2-7ae6-4c8d-ab15-6a8fbfffef48" containerName="manager" Jul 19 07:50:52.956574 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:52.956509 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="a86703d2-7ae6-4c8d-ab15-6a8fbfffef48" containerName="manager" Jul 19 07:50:52.956574 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:52.956535 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3dd100ae-9e01-4083-af93-87b3acc4d914" containerName="manager" Jul 19 07:50:52.956574 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:52.956540 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="3dd100ae-9e01-4083-af93-87b3acc4d914" containerName="manager" Jul 19 07:50:52.956677 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:52.956608 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="a86703d2-7ae6-4c8d-ab15-6a8fbfffef48" containerName="manager" Jul 19 07:50:52.956677 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:52.956617 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="3dd100ae-9e01-4083-af93-87b3acc4d914" containerName="manager" Jul 19 07:50:52.960131 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:52.960112 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ssftg" Jul 19 07:50:52.962675 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:50:52.962487 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-lkx5c\" is forbidden: User \"system:node:ip-10-0-136-70.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-70.ec2.internal' and this object" podUID="a86703d2-7ae6-4c8d-ab15-6a8fbfffef48" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lkx5c" Jul 19 07:50:52.965994 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:50:52.965947 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-qc5mf\" is forbidden: User \"system:node:ip-10-0-136-70.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-70.ec2.internal' and this object" podUID="3dd100ae-9e01-4083-af93-87b3acc4d914" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qc5mf" Jul 19 07:50:52.971114 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:52.971086 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ssftg"] Jul 19 07:50:53.008527 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:53.008470 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ht56p\" (UniqueName: \"kubernetes.io/projected/1d8ab2e7-2138-4419-8b7e-9f9ea75e0c8f-kube-api-access-ht56p\") pod \"kuadrant-operator-controller-manager-55467d9f7-t7dxv\" (UID: \"1d8ab2e7-2138-4419-8b7e-9f9ea75e0c8f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-t7dxv" Jul 19 07:50:53.008688 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:53.008610 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1d8ab2e7-2138-4419-8b7e-9f9ea75e0c8f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-t7dxv\" (UID: \"1d8ab2e7-2138-4419-8b7e-9f9ea75e0c8f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-t7dxv" Jul 19 07:50:53.091510 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:50:53.091478 2574 log.go:32] "RunPodSandbox from runtime service failed" err=< Jul 19 07:50:53.091510 ip-10-0-136-70 kubenswrapper[2574]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-55467d9f7-4nvz2_kuadrant-system_823547f7-9ebd-43d8-ad54-172d8c8c29f2_0(46f3e3b6f1c4dec87253d5bbdb1f28b560c60fe8b31d55e4a50a8e6820bd7a3a): error adding pod kuadrant-system_kuadrant-operator-controller-manager-55467d9f7-4nvz2 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"46f3e3b6f1c4dec87253d5bbdb1f28b560c60fe8b31d55e4a50a8e6820bd7a3a" Netns:"/var/run/netns/62bbd35d-dc2e-420c-89d2-b3f12ac17911" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=kuadrant-system;K8S_POD_NAME=kuadrant-operator-controller-manager-55467d9f7-4nvz2;K8S_POD_INFRA_CONTAINER_ID=46f3e3b6f1c4dec87253d5bbdb1f28b560c60fe8b31d55e4a50a8e6820bd7a3a;K8S_POD_UID=823547f7-9ebd-43d8-ad54-172d8c8c29f2" Path:"" ERRORED: error configuring pod [kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-4nvz2] networking: [kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-4nvz2/823547f7-9ebd-43d8-ad54-172d8c8c29f2:ovn-kubernetes]: error adding container to network "ovn-kubernetes": CNI request failed with status 400: '[kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-4nvz2 46f3e3b6f1c4dec87253d5bbdb1f28b560c60fe8b31d55e4a50a8e6820bd7a3a network default NAD default NAD key default] [kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-4nvz2 46f3e3b6f1c4dec87253d5bbdb1f28b560c60fe8b31d55e4a50a8e6820bd7a3a network default NAD default NAD key default] failed to configure pod interface: pod deleted waiting for OVS port binding for 0a:58:0a:84:00:25 [10.132.0.37/23] Jul 19 07:50:53.091510 ip-10-0-136-70 kubenswrapper[2574]: ' Jul 19 07:50:53.091510 ip-10-0-136-70 kubenswrapper[2574]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Jul 19 07:50:53.091510 ip-10-0-136-70 kubenswrapper[2574]: > Jul 19 07:50:53.091778 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:50:53.091555 2574 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=< Jul 19 07:50:53.091778 ip-10-0-136-70 kubenswrapper[2574]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-55467d9f7-4nvz2_kuadrant-system_823547f7-9ebd-43d8-ad54-172d8c8c29f2_0(46f3e3b6f1c4dec87253d5bbdb1f28b560c60fe8b31d55e4a50a8e6820bd7a3a): error adding pod kuadrant-system_kuadrant-operator-controller-manager-55467d9f7-4nvz2 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"46f3e3b6f1c4dec87253d5bbdb1f28b560c60fe8b31d55e4a50a8e6820bd7a3a" Netns:"/var/run/netns/62bbd35d-dc2e-420c-89d2-b3f12ac17911" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=kuadrant-system;K8S_POD_NAME=kuadrant-operator-controller-manager-55467d9f7-4nvz2;K8S_POD_INFRA_CONTAINER_ID=46f3e3b6f1c4dec87253d5bbdb1f28b560c60fe8b31d55e4a50a8e6820bd7a3a;K8S_POD_UID=823547f7-9ebd-43d8-ad54-172d8c8c29f2" Path:"" ERRORED: error configuring pod [kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-4nvz2] networking: [kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-4nvz2/823547f7-9ebd-43d8-ad54-172d8c8c29f2:ovn-kubernetes]: error adding container to network "ovn-kubernetes": CNI request failed with status 400: '[kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-4nvz2 46f3e3b6f1c4dec87253d5bbdb1f28b560c60fe8b31d55e4a50a8e6820bd7a3a network default NAD default NAD key default] [kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-4nvz2 46f3e3b6f1c4dec87253d5bbdb1f28b560c60fe8b31d55e4a50a8e6820bd7a3a network default NAD default NAD key default] failed to configure pod interface: pod deleted waiting for OVS port binding for 0a:58:0a:84:00:25 [10.132.0.37/23] Jul 19 07:50:53.091778 ip-10-0-136-70 kubenswrapper[2574]: ' Jul 19 07:50:53.091778 ip-10-0-136-70 kubenswrapper[2574]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Jul 19 07:50:53.091778 ip-10-0-136-70 kubenswrapper[2574]: > pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-4nvz2" Jul 19 07:50:53.110074 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:53.110027 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1d8ab2e7-2138-4419-8b7e-9f9ea75e0c8f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-t7dxv\" (UID: \"1d8ab2e7-2138-4419-8b7e-9f9ea75e0c8f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-t7dxv" Jul 19 07:50:53.110240 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:53.110112 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jmpw\" (UniqueName: \"kubernetes.io/projected/df18da6e-66ff-40d0-9957-3e82b79e14d4-kube-api-access-8jmpw\") pod \"limitador-operator-controller-manager-859d7b78bd-ssftg\" (UID: \"df18da6e-66ff-40d0-9957-3e82b79e14d4\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ssftg" Jul 19 07:50:53.110240 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:53.110226 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ht56p\" (UniqueName: \"kubernetes.io/projected/1d8ab2e7-2138-4419-8b7e-9f9ea75e0c8f-kube-api-access-ht56p\") pod \"kuadrant-operator-controller-manager-55467d9f7-t7dxv\" (UID: \"1d8ab2e7-2138-4419-8b7e-9f9ea75e0c8f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-t7dxv" Jul 19 07:50:53.110446 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:53.110427 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1d8ab2e7-2138-4419-8b7e-9f9ea75e0c8f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-t7dxv\" (UID: \"1d8ab2e7-2138-4419-8b7e-9f9ea75e0c8f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-t7dxv" Jul 19 07:50:53.119348 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:53.119291 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ht56p\" (UniqueName: \"kubernetes.io/projected/1d8ab2e7-2138-4419-8b7e-9f9ea75e0c8f-kube-api-access-ht56p\") pod \"kuadrant-operator-controller-manager-55467d9f7-t7dxv\" (UID: \"1d8ab2e7-2138-4419-8b7e-9f9ea75e0c8f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-t7dxv" Jul 19 07:50:53.165174 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:53.165147 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qc5mf" Jul 19 07:50:53.167578 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:53.167543 2574 generic.go:356] "Generic (PLEG): container finished" podID="a86703d2-7ae6-4c8d-ab15-6a8fbfffef48" containerID="521162a6d716530351a4acc02c45f98bb53c2eb320d5e8501ba2d21531cbf365" exitCode=0 Jul 19 07:50:53.167687 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:50:53.167570 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-lkx5c\" is forbidden: User \"system:node:ip-10-0-136-70.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-70.ec2.internal' and this object" podUID="a86703d2-7ae6-4c8d-ab15-6a8fbfffef48" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lkx5c" Jul 19 07:50:53.169244 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:53.169217 2574 generic.go:356] "Generic (PLEG): container finished" podID="3dd100ae-9e01-4083-af93-87b3acc4d914" containerID="bb0708b6a1a83ec6bce9d2fa144c0c10ebad096c918247f006fe0bb18d751033" exitCode=0 Jul 19 07:50:53.169352 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:53.169268 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qc5mf" Jul 19 07:50:53.169352 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:53.169276 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-4nvz2" Jul 19 07:50:53.169352 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:50:53.169316 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-qc5mf\" is forbidden: User \"system:node:ip-10-0-136-70.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-70.ec2.internal' and this object" podUID="3dd100ae-9e01-4083-af93-87b3acc4d914" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qc5mf" Jul 19 07:50:53.169352 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:53.169270 2574 scope.go:117] "RemoveContainer" containerID="bb0708b6a1a83ec6bce9d2fa144c0c10ebad096c918247f006fe0bb18d751033" Jul 19 07:50:53.171168 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:50:53.171133 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-lkx5c\" is forbidden: User \"system:node:ip-10-0-136-70.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-70.ec2.internal' and this object" podUID="a86703d2-7ae6-4c8d-ab15-6a8fbfffef48" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lkx5c" Jul 19 07:50:53.173198 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:50:53.173174 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-qc5mf\" is forbidden: User \"system:node:ip-10-0-136-70.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-70.ec2.internal' and this object" podUID="3dd100ae-9e01-4083-af93-87b3acc4d914" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qc5mf" Jul 19 07:50:53.173508 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:53.173493 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lkx5c" Jul 19 07:50:53.175025 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:50:53.174993 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-lkx5c\" is forbidden: User \"system:node:ip-10-0-136-70.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-70.ec2.internal' and this object" podUID="a86703d2-7ae6-4c8d-ab15-6a8fbfffef48" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lkx5c" Jul 19 07:50:53.176123 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:53.176108 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-4nvz2" Jul 19 07:50:53.176702 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:50:53.176678 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-qc5mf\" is forbidden: User \"system:node:ip-10-0-136-70.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-70.ec2.internal' and this object" podUID="3dd100ae-9e01-4083-af93-87b3acc4d914" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qc5mf" Jul 19 07:50:53.178670 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:50:53.178632 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-4nvz2\" is forbidden: User \"system:node:ip-10-0-136-70.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-70.ec2.internal' and this object" podUID="823547f7-9ebd-43d8-ad54-172d8c8c29f2" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-4nvz2" Jul 19 07:50:53.178727 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:53.178712 2574 scope.go:117] "RemoveContainer" containerID="bb0708b6a1a83ec6bce9d2fa144c0c10ebad096c918247f006fe0bb18d751033" Jul 19 07:50:53.178987 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:50:53.178969 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb0708b6a1a83ec6bce9d2fa144c0c10ebad096c918247f006fe0bb18d751033\": container with ID starting with bb0708b6a1a83ec6bce9d2fa144c0c10ebad096c918247f006fe0bb18d751033 not found: ID does not exist" containerID="bb0708b6a1a83ec6bce9d2fa144c0c10ebad096c918247f006fe0bb18d751033" Jul 19 07:50:53.179028 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:53.178995 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb0708b6a1a83ec6bce9d2fa144c0c10ebad096c918247f006fe0bb18d751033"} err="failed to get container status \"bb0708b6a1a83ec6bce9d2fa144c0c10ebad096c918247f006fe0bb18d751033\": rpc error: code = NotFound desc = could not find container \"bb0708b6a1a83ec6bce9d2fa144c0c10ebad096c918247f006fe0bb18d751033\": container with ID starting with bb0708b6a1a83ec6bce9d2fa144c0c10ebad096c918247f006fe0bb18d751033 not found: ID does not exist" Jul 19 07:50:53.180361 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:50:53.180342 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-lkx5c\" is forbidden: User \"system:node:ip-10-0-136-70.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-70.ec2.internal' and this object" podUID="a86703d2-7ae6-4c8d-ab15-6a8fbfffef48" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lkx5c" Jul 19 07:50:53.182309 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:50:53.182288 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-qc5mf\" is forbidden: User \"system:node:ip-10-0-136-70.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-70.ec2.internal' and this object" podUID="3dd100ae-9e01-4083-af93-87b3acc4d914" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qc5mf" Jul 19 07:50:53.184093 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:50:53.184075 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-4nvz2\" is forbidden: User \"system:node:ip-10-0-136-70.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-70.ec2.internal' and this object" podUID="823547f7-9ebd-43d8-ad54-172d8c8c29f2" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-4nvz2" Jul 19 07:50:53.207273 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:53.207243 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-t7dxv" Jul 19 07:50:53.211315 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:53.211289 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jmpw\" (UniqueName: \"kubernetes.io/projected/df18da6e-66ff-40d0-9957-3e82b79e14d4-kube-api-access-8jmpw\") pod \"limitador-operator-controller-manager-859d7b78bd-ssftg\" (UID: \"df18da6e-66ff-40d0-9957-3e82b79e14d4\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ssftg" Jul 19 07:50:53.219793 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:53.219757 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jmpw\" (UniqueName: \"kubernetes.io/projected/df18da6e-66ff-40d0-9957-3e82b79e14d4-kube-api-access-8jmpw\") pod \"limitador-operator-controller-manager-859d7b78bd-ssftg\" (UID: \"df18da6e-66ff-40d0-9957-3e82b79e14d4\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ssftg" Jul 19 07:50:53.311755 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:53.311681 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gpmvj\" (UniqueName: \"kubernetes.io/projected/a86703d2-7ae6-4c8d-ab15-6a8fbfffef48-kube-api-access-gpmvj\") pod \"a86703d2-7ae6-4c8d-ab15-6a8fbfffef48\" (UID: \"a86703d2-7ae6-4c8d-ab15-6a8fbfffef48\") " Jul 19 07:50:53.311755 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:53.311734 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3dd100ae-9e01-4083-af93-87b3acc4d914-extensions-socket-volume\") pod \"3dd100ae-9e01-4083-af93-87b3acc4d914\" (UID: \"3dd100ae-9e01-4083-af93-87b3acc4d914\") " Jul 19 07:50:53.311956 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:53.311772 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rsstq\" (UniqueName: \"kubernetes.io/projected/3dd100ae-9e01-4083-af93-87b3acc4d914-kube-api-access-rsstq\") pod \"3dd100ae-9e01-4083-af93-87b3acc4d914\" (UID: \"3dd100ae-9e01-4083-af93-87b3acc4d914\") " Jul 19 07:50:53.311956 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:53.311820 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/823547f7-9ebd-43d8-ad54-172d8c8c29f2-extensions-socket-volume\") pod \"823547f7-9ebd-43d8-ad54-172d8c8c29f2\" (UID: \"823547f7-9ebd-43d8-ad54-172d8c8c29f2\") " Jul 19 07:50:53.311956 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:53.311930 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l28jv\" (UniqueName: \"kubernetes.io/projected/823547f7-9ebd-43d8-ad54-172d8c8c29f2-kube-api-access-l28jv\") pod \"823547f7-9ebd-43d8-ad54-172d8c8c29f2\" (UID: \"823547f7-9ebd-43d8-ad54-172d8c8c29f2\") " Jul 19 07:50:53.312201 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:53.312161 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/823547f7-9ebd-43d8-ad54-172d8c8c29f2-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "823547f7-9ebd-43d8-ad54-172d8c8c29f2" (UID: "823547f7-9ebd-43d8-ad54-172d8c8c29f2"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 07:50:53.312404 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:53.312358 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dd100ae-9e01-4083-af93-87b3acc4d914-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "3dd100ae-9e01-4083-af93-87b3acc4d914" (UID: "3dd100ae-9e01-4083-af93-87b3acc4d914"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 07:50:53.314115 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:53.314087 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a86703d2-7ae6-4c8d-ab15-6a8fbfffef48-kube-api-access-gpmvj" (OuterVolumeSpecName: "kube-api-access-gpmvj") pod "a86703d2-7ae6-4c8d-ab15-6a8fbfffef48" (UID: "a86703d2-7ae6-4c8d-ab15-6a8fbfffef48"). InnerVolumeSpecName "kube-api-access-gpmvj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 07:50:53.314546 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:53.314479 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/823547f7-9ebd-43d8-ad54-172d8c8c29f2-kube-api-access-l28jv" (OuterVolumeSpecName: "kube-api-access-l28jv") pod "823547f7-9ebd-43d8-ad54-172d8c8c29f2" (UID: "823547f7-9ebd-43d8-ad54-172d8c8c29f2"). InnerVolumeSpecName "kube-api-access-l28jv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 07:50:53.314651 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:53.314582 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dd100ae-9e01-4083-af93-87b3acc4d914-kube-api-access-rsstq" (OuterVolumeSpecName: "kube-api-access-rsstq") pod "3dd100ae-9e01-4083-af93-87b3acc4d914" (UID: "3dd100ae-9e01-4083-af93-87b3acc4d914"). InnerVolumeSpecName "kube-api-access-rsstq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 07:50:53.339456 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:53.339434 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ssftg" Jul 19 07:50:53.353800 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:53.353774 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-t7dxv"] Jul 19 07:50:53.356195 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:50:53.356170 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d8ab2e7_2138_4419_8b7e_9f9ea75e0c8f.slice/crio-9ef411643300261427363386ebb8f35091befece50ab8a6d87db6ddd88a76c3a WatchSource:0}: Error finding container 9ef411643300261427363386ebb8f35091befece50ab8a6d87db6ddd88a76c3a: Status 404 returned error can't find the container with id 9ef411643300261427363386ebb8f35091befece50ab8a6d87db6ddd88a76c3a Jul 19 07:50:53.413086 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:53.413057 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l28jv\" (UniqueName: \"kubernetes.io/projected/823547f7-9ebd-43d8-ad54-172d8c8c29f2-kube-api-access-l28jv\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 07:50:53.413086 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:53.413085 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpmvj\" (UniqueName: \"kubernetes.io/projected/a86703d2-7ae6-4c8d-ab15-6a8fbfffef48-kube-api-access-gpmvj\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 07:50:53.413265 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:53.413095 2574 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3dd100ae-9e01-4083-af93-87b3acc4d914-extensions-socket-volume\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 07:50:53.413265 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:53.413104 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsstq\" (UniqueName: \"kubernetes.io/projected/3dd100ae-9e01-4083-af93-87b3acc4d914-kube-api-access-rsstq\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 07:50:53.413265 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:53.413112 2574 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/823547f7-9ebd-43d8-ad54-172d8c8c29f2-extensions-socket-volume\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 07:50:53.479668 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:50:53.479638 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-4nvz2\" is forbidden: User \"system:node:ip-10-0-136-70.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-70.ec2.internal' and this object" podUID="823547f7-9ebd-43d8-ad54-172d8c8c29f2" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-4nvz2" Jul 19 07:50:53.480269 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:53.480246 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ssftg"] Jul 19 07:50:53.481726 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:50:53.481704 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-lkx5c\" is forbidden: User \"system:node:ip-10-0-136-70.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-70.ec2.internal' and this object" podUID="a86703d2-7ae6-4c8d-ab15-6a8fbfffef48" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lkx5c" Jul 19 07:50:53.482248 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:50:53.482225 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf18da6e_66ff_40d0_9957_3e82b79e14d4.slice/crio-6f83c56fd35c8fb01930c4ee24158f85f81ce11b098fa83e9a486c8cb2b3d193 WatchSource:0}: Error finding container 6f83c56fd35c8fb01930c4ee24158f85f81ce11b098fa83e9a486c8cb2b3d193: Status 404 returned error can't find the container with id 6f83c56fd35c8fb01930c4ee24158f85f81ce11b098fa83e9a486c8cb2b3d193 Jul 19 07:50:53.483540 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:50:53.483496 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-qc5mf\" is forbidden: User \"system:node:ip-10-0-136-70.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-70.ec2.internal' and this object" podUID="3dd100ae-9e01-4083-af93-87b3acc4d914" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qc5mf" Jul 19 07:50:53.647028 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:53.646944 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:50:53.752236 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:53.752187 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:50:54.176884 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:54.176843 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-t7dxv" event={"ID":"1d8ab2e7-2138-4419-8b7e-9f9ea75e0c8f","Type":"ContainerStarted","Data":"af914c967cc93be898c2a0f69a8bdadb3100c6886695a69527dfafbfe52d3605"} Jul 19 07:50:54.177105 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:54.176913 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-t7dxv" event={"ID":"1d8ab2e7-2138-4419-8b7e-9f9ea75e0c8f","Type":"ContainerStarted","Data":"9ef411643300261427363386ebb8f35091befece50ab8a6d87db6ddd88a76c3a"} Jul 19 07:50:54.177105 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:54.177044 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-t7dxv" Jul 19 07:50:54.179140 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:50:54.179112 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-qc5mf\" is forbidden: User \"system:node:ip-10-0-136-70.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-70.ec2.internal' and this object" podUID="3dd100ae-9e01-4083-af93-87b3acc4d914" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qc5mf" Jul 19 07:50:54.179285 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:54.179143 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lkx5c" Jul 19 07:50:54.179285 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:54.179156 2574 scope.go:117] "RemoveContainer" containerID="521162a6d716530351a4acc02c45f98bb53c2eb320d5e8501ba2d21531cbf365" Jul 19 07:50:54.180850 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:54.180821 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ssftg" event={"ID":"df18da6e-66ff-40d0-9957-3e82b79e14d4","Type":"ContainerStarted","Data":"29531da2a21d918fe7d4fba71ebb87609c8f08abfb1785f80d55b4311875480f"} Jul 19 07:50:54.180963 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:54.180875 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-4nvz2" Jul 19 07:50:54.181018 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:54.180874 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ssftg" event={"ID":"df18da6e-66ff-40d0-9957-3e82b79e14d4","Type":"ContainerStarted","Data":"6f83c56fd35c8fb01930c4ee24158f85f81ce11b098fa83e9a486c8cb2b3d193"} Jul 19 07:50:54.181054 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:54.181020 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ssftg" Jul 19 07:50:54.181097 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:50:54.181048 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-4nvz2\" is forbidden: User \"system:node:ip-10-0-136-70.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-70.ec2.internal' and this object" podUID="823547f7-9ebd-43d8-ad54-172d8c8c29f2" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-4nvz2" Jul 19 07:50:54.200641 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:54.200586 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-t7dxv" podStartSLOduration=2.200567599 podStartE2EDuration="2.200567599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 07:50:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:50:54.199144934 +0000 UTC m=+520.553454534" watchObservedRunningTime="2026-07-19 07:50:54.200567599 +0000 UTC m=+520.554877203" Jul 19 07:50:54.201570 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:50:54.201537 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-lkx5c\" is forbidden: User \"system:node:ip-10-0-136-70.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-70.ec2.internal' and this object" podUID="a86703d2-7ae6-4c8d-ab15-6a8fbfffef48" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lkx5c" Jul 19 07:50:54.203428 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:50:54.203406 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-4nvz2\" is forbidden: User \"system:node:ip-10-0-136-70.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-70.ec2.internal' and this object" podUID="823547f7-9ebd-43d8-ad54-172d8c8c29f2" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-4nvz2" Jul 19 07:50:54.205348 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:50:54.205326 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-lkx5c\" is forbidden: User \"system:node:ip-10-0-136-70.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-70.ec2.internal' and this object" podUID="a86703d2-7ae6-4c8d-ab15-6a8fbfffef48" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lkx5c" Jul 19 07:50:54.207187 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:50:54.207169 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-qc5mf\" is forbidden: User \"system:node:ip-10-0-136-70.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-70.ec2.internal' and this object" podUID="3dd100ae-9e01-4083-af93-87b3acc4d914" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qc5mf" Jul 19 07:50:54.229571 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:54.229448 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ssftg" podStartSLOduration=2.22942936 podStartE2EDuration="2.22942936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 07:50:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:50:54.228949257 +0000 UTC m=+520.583258859" watchObservedRunningTime="2026-07-19 07:50:54.22942936 +0000 UTC m=+520.583738962" Jul 19 07:50:54.556967 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:50:54.556926 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-4nvz2\" is forbidden: User \"system:node:ip-10-0-136-70.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-70.ec2.internal' and this object" podUID="823547f7-9ebd-43d8-ad54-172d8c8c29f2" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-4nvz2" Jul 19 07:50:54.558724 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:50:54.558695 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-lkx5c\" is forbidden: User \"system:node:ip-10-0-136-70.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-70.ec2.internal' and this object" podUID="a86703d2-7ae6-4c8d-ab15-6a8fbfffef48" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lkx5c" Jul 19 07:50:54.559697 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:54.559669 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3dd100ae-9e01-4083-af93-87b3acc4d914" path="/var/lib/kubelet/pods/3dd100ae-9e01-4083-af93-87b3acc4d914/volumes" Jul 19 07:50:54.560116 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:54.560095 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="823547f7-9ebd-43d8-ad54-172d8c8c29f2" path="/var/lib/kubelet/pods/823547f7-9ebd-43d8-ad54-172d8c8c29f2/volumes" Jul 19 07:50:54.560347 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:54.560330 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a86703d2-7ae6-4c8d-ab15-6a8fbfffef48" path="/var/lib/kubelet/pods/a86703d2-7ae6-4c8d-ab15-6a8fbfffef48/volumes" Jul 19 07:50:54.560417 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:50:54.560367 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-qc5mf\" is forbidden: User \"system:node:ip-10-0-136-70.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-70.ec2.internal' and this object" podUID="3dd100ae-9e01-4083-af93-87b3acc4d914" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qc5mf" Jul 19 07:50:56.650111 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:56.650057 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:50:56.750720 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:56.750680 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:50:59.253299 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:59.253265 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:50:59.352890 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:50:59.352857 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:51:02.847057 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:51:02.847023 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:51:02.902072 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:51:02.902031 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:51:05.192308 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:51:05.192274 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-t7dxv" Jul 19 07:51:05.192737 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:51:05.192478 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ssftg" Jul 19 07:51:05.514582 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:51:05.514545 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-szh92"] Jul 19 07:51:05.521211 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:51:05.521187 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-szh92" Jul 19 07:51:05.534233 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:51:05.534207 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-szh92"] Jul 19 07:51:05.612394 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:51:05.612355 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nstz\" (UniqueName: \"kubernetes.io/projected/438041f9-2e00-4d37-9d04-2ab07314e1dc-kube-api-access-4nstz\") pod \"kuadrant-operator-controller-manager-86878c769f-szh92\" (UID: \"438041f9-2e00-4d37-9d04-2ab07314e1dc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-szh92" Jul 19 07:51:05.612394 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:51:05.612395 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/438041f9-2e00-4d37-9d04-2ab07314e1dc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-szh92\" (UID: \"438041f9-2e00-4d37-9d04-2ab07314e1dc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-szh92" Jul 19 07:51:05.713392 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:51:05.713363 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nstz\" (UniqueName: \"kubernetes.io/projected/438041f9-2e00-4d37-9d04-2ab07314e1dc-kube-api-access-4nstz\") pod \"kuadrant-operator-controller-manager-86878c769f-szh92\" (UID: \"438041f9-2e00-4d37-9d04-2ab07314e1dc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-szh92" Jul 19 07:51:05.713584 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:51:05.713403 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/438041f9-2e00-4d37-9d04-2ab07314e1dc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-szh92\" (UID: \"438041f9-2e00-4d37-9d04-2ab07314e1dc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-szh92" Jul 19 07:51:05.713778 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:51:05.713762 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/438041f9-2e00-4d37-9d04-2ab07314e1dc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-szh92\" (UID: \"438041f9-2e00-4d37-9d04-2ab07314e1dc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-szh92" Jul 19 07:51:05.724609 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:51:05.724576 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nstz\" (UniqueName: \"kubernetes.io/projected/438041f9-2e00-4d37-9d04-2ab07314e1dc-kube-api-access-4nstz\") pod \"kuadrant-operator-controller-manager-86878c769f-szh92\" (UID: \"438041f9-2e00-4d37-9d04-2ab07314e1dc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-szh92" Jul 19 07:51:05.833910 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:51:05.833876 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-szh92" Jul 19 07:51:05.972204 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:51:05.972170 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-szh92"] Jul 19 07:51:05.975168 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:51:05.975136 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod438041f9_2e00_4d37_9d04_2ab07314e1dc.slice/crio-ac17b124e11fd62d7f4003099aea8763d0dbd3ae72efec9cdaa18294e10a8fb8 WatchSource:0}: Error finding container ac17b124e11fd62d7f4003099aea8763d0dbd3ae72efec9cdaa18294e10a8fb8: Status 404 returned error can't find the container with id ac17b124e11fd62d7f4003099aea8763d0dbd3ae72efec9cdaa18294e10a8fb8 Jul 19 07:51:06.256186 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:51:06.256082 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-szh92" event={"ID":"438041f9-2e00-4d37-9d04-2ab07314e1dc","Type":"ContainerStarted","Data":"3ca6f976d1b43f233b02622c6f1a13a8eb610f7cf5e948500350db2405619373"} Jul 19 07:51:06.256722 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:51:06.256193 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-szh92" event={"ID":"438041f9-2e00-4d37-9d04-2ab07314e1dc","Type":"ContainerStarted","Data":"ac17b124e11fd62d7f4003099aea8763d0dbd3ae72efec9cdaa18294e10a8fb8"} Jul 19 07:51:06.256722 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:51:06.256444 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-szh92" Jul 19 07:51:06.293829 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:51:06.293771 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-szh92" podStartSLOduration=1.293752051 podStartE2EDuration="1.293752051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 07:51:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:51:06.291373474 +0000 UTC m=+532.645683075" watchObservedRunningTime="2026-07-19 07:51:06.293752051 +0000 UTC m=+532.648061652" Jul 19 07:51:12.061117 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:51:12.061084 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:51:12.160634 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:51:12.160592 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:51:15.053032 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:51:15.052991 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:51:15.169713 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:51:15.169670 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:51:17.264514 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:51:17.264481 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-szh92" Jul 19 07:51:17.310748 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:51:17.310711 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-t7dxv"] Jul 19 07:51:17.311012 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:51:17.310981 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-t7dxv" podUID="1d8ab2e7-2138-4419-8b7e-9f9ea75e0c8f" containerName="manager" containerID="cri-o://af914c967cc93be898c2a0f69a8bdadb3100c6886695a69527dfafbfe52d3605" gracePeriod=10 Jul 19 07:51:17.563894 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:51:17.563859 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-t7dxv" Jul 19 07:51:17.697064 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:51:17.697009 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1d8ab2e7-2138-4419-8b7e-9f9ea75e0c8f-extensions-socket-volume\") pod \"1d8ab2e7-2138-4419-8b7e-9f9ea75e0c8f\" (UID: \"1d8ab2e7-2138-4419-8b7e-9f9ea75e0c8f\") " Jul 19 07:51:17.697257 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:51:17.697097 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ht56p\" (UniqueName: \"kubernetes.io/projected/1d8ab2e7-2138-4419-8b7e-9f9ea75e0c8f-kube-api-access-ht56p\") pod \"1d8ab2e7-2138-4419-8b7e-9f9ea75e0c8f\" (UID: \"1d8ab2e7-2138-4419-8b7e-9f9ea75e0c8f\") " Jul 19 07:51:17.697426 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:51:17.697402 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d8ab2e7-2138-4419-8b7e-9f9ea75e0c8f-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "1d8ab2e7-2138-4419-8b7e-9f9ea75e0c8f" (UID: "1d8ab2e7-2138-4419-8b7e-9f9ea75e0c8f"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 07:51:17.699183 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:51:17.699159 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d8ab2e7-2138-4419-8b7e-9f9ea75e0c8f-kube-api-access-ht56p" (OuterVolumeSpecName: "kube-api-access-ht56p") pod "1d8ab2e7-2138-4419-8b7e-9f9ea75e0c8f" (UID: "1d8ab2e7-2138-4419-8b7e-9f9ea75e0c8f"). InnerVolumeSpecName "kube-api-access-ht56p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 07:51:17.797839 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:51:17.797737 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ht56p\" (UniqueName: \"kubernetes.io/projected/1d8ab2e7-2138-4419-8b7e-9f9ea75e0c8f-kube-api-access-ht56p\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 07:51:17.797839 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:51:17.797781 2574 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1d8ab2e7-2138-4419-8b7e-9f9ea75e0c8f-extensions-socket-volume\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 07:51:18.316295 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:51:18.316250 2574 generic.go:356] "Generic (PLEG): container finished" podID="1d8ab2e7-2138-4419-8b7e-9f9ea75e0c8f" containerID="af914c967cc93be898c2a0f69a8bdadb3100c6886695a69527dfafbfe52d3605" exitCode=0 Jul 19 07:51:18.316830 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:51:18.316317 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-t7dxv" Jul 19 07:51:18.316830 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:51:18.316329 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-t7dxv" event={"ID":"1d8ab2e7-2138-4419-8b7e-9f9ea75e0c8f","Type":"ContainerDied","Data":"af914c967cc93be898c2a0f69a8bdadb3100c6886695a69527dfafbfe52d3605"} Jul 19 07:51:18.316830 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:51:18.316407 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-t7dxv" event={"ID":"1d8ab2e7-2138-4419-8b7e-9f9ea75e0c8f","Type":"ContainerDied","Data":"9ef411643300261427363386ebb8f35091befece50ab8a6d87db6ddd88a76c3a"} Jul 19 07:51:18.316830 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:51:18.316431 2574 scope.go:117] "RemoveContainer" containerID="af914c967cc93be898c2a0f69a8bdadb3100c6886695a69527dfafbfe52d3605" Jul 19 07:51:18.325867 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:51:18.325849 2574 scope.go:117] "RemoveContainer" containerID="af914c967cc93be898c2a0f69a8bdadb3100c6886695a69527dfafbfe52d3605" Jul 19 07:51:18.326120 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:51:18.326102 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af914c967cc93be898c2a0f69a8bdadb3100c6886695a69527dfafbfe52d3605\": container with ID starting with af914c967cc93be898c2a0f69a8bdadb3100c6886695a69527dfafbfe52d3605 not found: ID does not exist" containerID="af914c967cc93be898c2a0f69a8bdadb3100c6886695a69527dfafbfe52d3605" Jul 19 07:51:18.326180 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:51:18.326128 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af914c967cc93be898c2a0f69a8bdadb3100c6886695a69527dfafbfe52d3605"} err="failed to get container status \"af914c967cc93be898c2a0f69a8bdadb3100c6886695a69527dfafbfe52d3605\": rpc error: code = NotFound desc = could not find container \"af914c967cc93be898c2a0f69a8bdadb3100c6886695a69527dfafbfe52d3605\": container with ID starting with af914c967cc93be898c2a0f69a8bdadb3100c6886695a69527dfafbfe52d3605 not found: ID does not exist" Jul 19 07:51:18.338476 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:51:18.338449 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-t7dxv"] Jul 19 07:51:18.344149 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:51:18.344124 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-t7dxv"] Jul 19 07:51:18.561109 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:51:18.561076 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d8ab2e7-2138-4419-8b7e-9f9ea75e0c8f" path="/var/lib/kubelet/pods/1d8ab2e7-2138-4419-8b7e-9f9ea75e0c8f/volumes" Jul 19 07:51:21.560997 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:51:21.560955 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:51:21.634343 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:51:21.634298 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:51:37.409465 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:51:37.409432 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:51:37.461503 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:51:37.461453 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:51:38.051450 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:51:38.051416 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:51:38.165811 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:51:38.165747 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:51:38.214664 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:51:38.214619 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:51:38.462711 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:51:38.462676 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 07:51:38.463191 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:51:38.463169 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d8ab2e7-2138-4419-8b7e-9f9ea75e0c8f" containerName="manager" Jul 19 07:51:38.463301 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:51:38.463193 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d8ab2e7-2138-4419-8b7e-9f9ea75e0c8f" containerName="manager" Jul 19 07:51:38.463301 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:51:38.463297 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d8ab2e7-2138-4419-8b7e-9f9ea75e0c8f" containerName="manager" Jul 19 07:51:38.466381 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:51:38.466364 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-tgs9r" Jul 19 07:51:38.468753 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:51:38.468731 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-k4wmr\"" Jul 19 07:51:38.468849 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:51:38.468731 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 19 07:51:38.473279 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:51:38.473253 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 07:51:38.490226 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:51:38.490198 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 07:51:38.553443 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:51:38.553410 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqcpd\" (UniqueName: \"kubernetes.io/projected/78f72907-2f0e-487c-b31e-70e2a8727400-kube-api-access-nqcpd\") pod \"limitador-limitador-54d6d988d5-tgs9r\" (UID: \"78f72907-2f0e-487c-b31e-70e2a8727400\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-tgs9r" Jul 19 07:51:38.553650 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:51:38.553455 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/78f72907-2f0e-487c-b31e-70e2a8727400-config-file\") pod \"limitador-limitador-54d6d988d5-tgs9r\" (UID: \"78f72907-2f0e-487c-b31e-70e2a8727400\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-tgs9r" Jul 19 07:51:38.655028 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:51:38.654989 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqcpd\" (UniqueName: \"kubernetes.io/projected/78f72907-2f0e-487c-b31e-70e2a8727400-kube-api-access-nqcpd\") pod \"limitador-limitador-54d6d988d5-tgs9r\" (UID: \"78f72907-2f0e-487c-b31e-70e2a8727400\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-tgs9r" Jul 19 07:51:38.655229 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:51:38.655037 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/78f72907-2f0e-487c-b31e-70e2a8727400-config-file\") pod \"limitador-limitador-54d6d988d5-tgs9r\" (UID: \"78f72907-2f0e-487c-b31e-70e2a8727400\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-tgs9r" Jul 19 07:51:38.655757 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:51:38.655731 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/78f72907-2f0e-487c-b31e-70e2a8727400-config-file\") pod \"limitador-limitador-54d6d988d5-tgs9r\" (UID: \"78f72907-2f0e-487c-b31e-70e2a8727400\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-tgs9r" Jul 19 07:51:38.662544 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:51:38.662504 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqcpd\" (UniqueName: \"kubernetes.io/projected/78f72907-2f0e-487c-b31e-70e2a8727400-kube-api-access-nqcpd\") pod \"limitador-limitador-54d6d988d5-tgs9r\" (UID: \"78f72907-2f0e-487c-b31e-70e2a8727400\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-tgs9r" Jul 19 07:51:38.777767 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:51:38.777677 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-tgs9r" Jul 19 07:51:38.896172 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:51:38.896121 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:51:38.923317 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:51:38.923289 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 07:51:38.925954 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:51:38.925922 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78f72907_2f0e_487c_b31e_70e2a8727400.slice/crio-83a3777a727772721cf9e742a6ec5afb8bec5449a9ebc849fdad832c8d861496 WatchSource:0}: Error finding container 83a3777a727772721cf9e742a6ec5afb8bec5449a9ebc849fdad832c8d861496: Status 404 returned error can't find the container with id 83a3777a727772721cf9e742a6ec5afb8bec5449a9ebc849fdad832c8d861496 Jul 19 07:51:38.996229 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:51:38.996192 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:51:39.068828 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:51:39.068735 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-2mjmd"] Jul 19 07:51:39.073350 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:51:39.073324 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-2mjmd" Jul 19 07:51:39.075776 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:51:39.075756 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-bqx2c\"" Jul 19 07:51:39.079089 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:51:39.079067 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-2mjmd"] Jul 19 07:51:39.159303 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:51:39.159258 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkl8l\" (UniqueName: \"kubernetes.io/projected/615e2145-0f67-436d-8dfa-50049bec3059-kube-api-access-dkl8l\") pod \"authorino-75d556bd77-2mjmd\" (UID: \"615e2145-0f67-436d-8dfa-50049bec3059\") " pod="kuadrant-system/authorino-75d556bd77-2mjmd" Jul 19 07:51:39.260709 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:51:39.260674 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkl8l\" (UniqueName: \"kubernetes.io/projected/615e2145-0f67-436d-8dfa-50049bec3059-kube-api-access-dkl8l\") pod \"authorino-75d556bd77-2mjmd\" (UID: \"615e2145-0f67-436d-8dfa-50049bec3059\") " pod="kuadrant-system/authorino-75d556bd77-2mjmd" Jul 19 07:51:39.269555 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:51:39.269502 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkl8l\" (UniqueName: \"kubernetes.io/projected/615e2145-0f67-436d-8dfa-50049bec3059-kube-api-access-dkl8l\") pod \"authorino-75d556bd77-2mjmd\" (UID: \"615e2145-0f67-436d-8dfa-50049bec3059\") " pod="kuadrant-system/authorino-75d556bd77-2mjmd" Jul 19 07:51:39.384704 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:51:39.384615 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-2mjmd" Jul 19 07:51:39.422306 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:51:39.422260 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-tgs9r" event={"ID":"78f72907-2f0e-487c-b31e-70e2a8727400","Type":"ContainerStarted","Data":"83a3777a727772721cf9e742a6ec5afb8bec5449a9ebc849fdad832c8d861496"} Jul 19 07:51:39.551447 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:51:39.551271 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-2mjmd"] Jul 19 07:51:40.434740 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:51:40.434697 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-2mjmd" event={"ID":"615e2145-0f67-436d-8dfa-50049bec3059","Type":"ContainerStarted","Data":"55228ff8d102d7f0a274fa757bd1d620fbd08253ad9099f5713f2359ab81d7a2"} Jul 19 07:51:44.456904 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:51:44.456849 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-2mjmd" event={"ID":"615e2145-0f67-436d-8dfa-50049bec3059","Type":"ContainerStarted","Data":"e4db6fbba246c9194d0290745c73c1aca101c10cdd2bdc0898fb42e080211d9b"} Jul 19 07:51:44.458632 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:51:44.458593 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-tgs9r" event={"ID":"78f72907-2f0e-487c-b31e-70e2a8727400","Type":"ContainerStarted","Data":"0b3e0f4736cd17399407ab266acca28289c8f92b6debb98a92155a5279e96fb2"} Jul 19 07:51:44.458789 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:51:44.458722 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-tgs9r" Jul 19 07:51:44.470885 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:51:44.470839 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-2mjmd" podStartSLOduration=1.598906039 podStartE2EDuration="5.470823763s" totalImagesPullingTime="3.871917724s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 07:51:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:51:44.469680431 +0000 UTC m=+570.823990033" watchObservedRunningTime="2026-07-19 07:51:44.470823763 +0000 UTC m=+570.825133363" Jul 19 07:51:44.485965 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:51:44.485908 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-tgs9r" podStartSLOduration=1.992532832 podStartE2EDuration="6.485892482s" totalImagesPullingTime="4.49335965s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 07:51:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:51:44.484006822 +0000 UTC m=+570.838316424" watchObservedRunningTime="2026-07-19 07:51:44.485892482 +0000 UTC m=+570.840202113" Jul 19 07:51:55.465410 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:51:55.465376 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-tgs9r" Jul 19 07:52:00.509632 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:52:00.509587 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:52:00.608173 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:52:00.608137 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:52:00.972372 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:52:00.972338 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:52:12.520498 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:52:12.520457 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:52:12.748529 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:52:12.748480 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-rf2r2"] Jul 19 07:52:12.753892 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:52:12.753871 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-rf2r2" Jul 19 07:52:12.760973 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:52:12.760945 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-rf2r2"] Jul 19 07:52:12.864897 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:52:12.864803 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qb4qj\" (UniqueName: \"kubernetes.io/projected/bd8008ce-62d2-4372-a3a5-e64261c2be59-kube-api-access-qb4qj\") pod \"authorino-5d4b67c446-rf2r2\" (UID: \"bd8008ce-62d2-4372-a3a5-e64261c2be59\") " pod="kuadrant-system/authorino-5d4b67c446-rf2r2" Jul 19 07:52:12.889459 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:52:12.889418 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:52:12.965935 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:52:12.965899 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qb4qj\" (UniqueName: \"kubernetes.io/projected/bd8008ce-62d2-4372-a3a5-e64261c2be59-kube-api-access-qb4qj\") pod \"authorino-5d4b67c446-rf2r2\" (UID: \"bd8008ce-62d2-4372-a3a5-e64261c2be59\") " pod="kuadrant-system/authorino-5d4b67c446-rf2r2" Jul 19 07:52:12.973707 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:52:12.973675 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qb4qj\" (UniqueName: \"kubernetes.io/projected/bd8008ce-62d2-4372-a3a5-e64261c2be59-kube-api-access-qb4qj\") pod \"authorino-5d4b67c446-rf2r2\" (UID: \"bd8008ce-62d2-4372-a3a5-e64261c2be59\") " pod="kuadrant-system/authorino-5d4b67c446-rf2r2" Jul 19 07:52:12.976055 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:52:12.976025 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-rf2r2"] Jul 19 07:52:12.976319 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:52:12.976302 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-rf2r2" Jul 19 07:52:13.001559 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:52:13.001527 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-568d44cffc-nf2t8"] Jul 19 07:52:13.004889 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:52:13.004866 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-568d44cffc-nf2t8" Jul 19 07:52:13.012413 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:52:13.012379 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-568d44cffc-nf2t8"] Jul 19 07:52:13.096426 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:52:13.096393 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-568d44cffc-nf2t8"] Jul 19 07:52:13.096717 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:52:13.096698 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-sjgg6], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-568d44cffc-nf2t8" podUID="032f007f-b7c8-4baf-8c6e-0305196413fd" Jul 19 07:52:13.130839 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:52:13.130750 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-c4f6ccf57-kphwt"] Jul 19 07:52:13.134027 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:52:13.134004 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-rf2r2"] Jul 19 07:52:13.134146 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:52:13.134018 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbd8008ce_62d2_4372_a3a5_e64261c2be59.slice/crio-7562f72ac703964c672019c521baef0e6661344a35b9323a583daee2bb2e8cac WatchSource:0}: Error finding container 7562f72ac703964c672019c521baef0e6661344a35b9323a583daee2bb2e8cac: Status 404 returned error can't find the container with id 7562f72ac703964c672019c521baef0e6661344a35b9323a583daee2bb2e8cac Jul 19 07:52:13.134146 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:52:13.134095 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c4f6ccf57-kphwt" Jul 19 07:52:13.136351 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:52:13.136328 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 19 07:52:13.140616 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:52:13.140593 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c4f6ccf57-kphwt"] Jul 19 07:52:13.168262 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:52:13.168230 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjgg6\" (UniqueName: \"kubernetes.io/projected/032f007f-b7c8-4baf-8c6e-0305196413fd-kube-api-access-sjgg6\") pod \"authorino-568d44cffc-nf2t8\" (UID: \"032f007f-b7c8-4baf-8c6e-0305196413fd\") " pod="kuadrant-system/authorino-568d44cffc-nf2t8" Jul 19 07:52:13.268970 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:52:13.268936 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjgg6\" (UniqueName: \"kubernetes.io/projected/032f007f-b7c8-4baf-8c6e-0305196413fd-kube-api-access-sjgg6\") pod \"authorino-568d44cffc-nf2t8\" (UID: \"032f007f-b7c8-4baf-8c6e-0305196413fd\") " pod="kuadrant-system/authorino-568d44cffc-nf2t8" Jul 19 07:52:13.268970 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:52:13.268971 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f094d7c7-811b-4efe-86a8-c4b586b38260-tls-cert\") pod \"authorino-c4f6ccf57-kphwt\" (UID: \"f094d7c7-811b-4efe-86a8-c4b586b38260\") " pod="kuadrant-system/authorino-c4f6ccf57-kphwt" Jul 19 07:52:13.269191 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:52:13.269020 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnjhb\" (UniqueName: \"kubernetes.io/projected/f094d7c7-811b-4efe-86a8-c4b586b38260-kube-api-access-nnjhb\") pod \"authorino-c4f6ccf57-kphwt\" (UID: \"f094d7c7-811b-4efe-86a8-c4b586b38260\") " pod="kuadrant-system/authorino-c4f6ccf57-kphwt" Jul 19 07:52:13.276791 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:52:13.276764 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjgg6\" (UniqueName: \"kubernetes.io/projected/032f007f-b7c8-4baf-8c6e-0305196413fd-kube-api-access-sjgg6\") pod \"authorino-568d44cffc-nf2t8\" (UID: \"032f007f-b7c8-4baf-8c6e-0305196413fd\") " pod="kuadrant-system/authorino-568d44cffc-nf2t8" Jul 19 07:52:13.370498 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:52:13.370453 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f094d7c7-811b-4efe-86a8-c4b586b38260-tls-cert\") pod \"authorino-c4f6ccf57-kphwt\" (UID: \"f094d7c7-811b-4efe-86a8-c4b586b38260\") " pod="kuadrant-system/authorino-c4f6ccf57-kphwt" Jul 19 07:52:13.370718 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:52:13.370580 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnjhb\" (UniqueName: \"kubernetes.io/projected/f094d7c7-811b-4efe-86a8-c4b586b38260-kube-api-access-nnjhb\") pod \"authorino-c4f6ccf57-kphwt\" (UID: \"f094d7c7-811b-4efe-86a8-c4b586b38260\") " pod="kuadrant-system/authorino-c4f6ccf57-kphwt" Jul 19 07:52:13.372979 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:52:13.372954 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f094d7c7-811b-4efe-86a8-c4b586b38260-tls-cert\") pod \"authorino-c4f6ccf57-kphwt\" (UID: \"f094d7c7-811b-4efe-86a8-c4b586b38260\") " pod="kuadrant-system/authorino-c4f6ccf57-kphwt" Jul 19 07:52:13.377349 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:52:13.377325 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnjhb\" (UniqueName: \"kubernetes.io/projected/f094d7c7-811b-4efe-86a8-c4b586b38260-kube-api-access-nnjhb\") pod \"authorino-c4f6ccf57-kphwt\" (UID: \"f094d7c7-811b-4efe-86a8-c4b586b38260\") " pod="kuadrant-system/authorino-c4f6ccf57-kphwt" Jul 19 07:52:13.453169 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:52:13.453097 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c4f6ccf57-kphwt" Jul 19 07:52:13.600156 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:52:13.600128 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-568d44cffc-nf2t8" Jul 19 07:52:13.600576 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:52:13.600129 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-rf2r2" event={"ID":"bd8008ce-62d2-4372-a3a5-e64261c2be59","Type":"ContainerStarted","Data":"7562f72ac703964c672019c521baef0e6661344a35b9323a583daee2bb2e8cac"} Jul 19 07:52:13.608504 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:52:13.608479 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-568d44cffc-nf2t8" Jul 19 07:52:13.618465 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:52:13.618440 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c4f6ccf57-kphwt"] Jul 19 07:52:13.622582 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:52:13.622558 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf094d7c7_811b_4efe_86a8_c4b586b38260.slice/crio-fd785535c4a5a144bbd3b839a35d43ae1b9c12a9f30d1260b636eefc5f7cc09a WatchSource:0}: Error finding container fd785535c4a5a144bbd3b839a35d43ae1b9c12a9f30d1260b636eefc5f7cc09a: Status 404 returned error can't find the container with id fd785535c4a5a144bbd3b839a35d43ae1b9c12a9f30d1260b636eefc5f7cc09a Jul 19 07:52:13.774432 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:52:13.774389 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sjgg6\" (UniqueName: \"kubernetes.io/projected/032f007f-b7c8-4baf-8c6e-0305196413fd-kube-api-access-sjgg6\") pod \"032f007f-b7c8-4baf-8c6e-0305196413fd\" (UID: \"032f007f-b7c8-4baf-8c6e-0305196413fd\") " Jul 19 07:52:13.776525 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:52:13.776495 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/032f007f-b7c8-4baf-8c6e-0305196413fd-kube-api-access-sjgg6" (OuterVolumeSpecName: "kube-api-access-sjgg6") pod "032f007f-b7c8-4baf-8c6e-0305196413fd" (UID: "032f007f-b7c8-4baf-8c6e-0305196413fd"). InnerVolumeSpecName "kube-api-access-sjgg6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 07:52:13.875507 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:52:13.875476 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjgg6\" (UniqueName: \"kubernetes.io/projected/032f007f-b7c8-4baf-8c6e-0305196413fd-kube-api-access-sjgg6\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 07:52:14.491846 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:52:14.491812 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wmhfd_dd68d31d-f763-433a-867b-65b12eabac76/console-operator/1.log" Jul 19 07:52:14.498779 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:52:14.498755 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wmhfd_dd68d31d-f763-433a-867b-65b12eabac76/console-operator/1.log" Jul 19 07:52:14.605783 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:52:14.605750 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-rf2r2" event={"ID":"bd8008ce-62d2-4372-a3a5-e64261c2be59","Type":"ContainerStarted","Data":"00fba717a91869a738d496da86a8fdb4b4c2187dc8ba2dfd3f71c6f3871ed528"} Jul 19 07:52:14.606233 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:52:14.605801 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-rf2r2" podUID="bd8008ce-62d2-4372-a3a5-e64261c2be59" containerName="authorino" containerID="cri-o://00fba717a91869a738d496da86a8fdb4b4c2187dc8ba2dfd3f71c6f3871ed528" gracePeriod=30 Jul 19 07:52:14.607366 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:52:14.607336 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c4f6ccf57-kphwt" event={"ID":"f094d7c7-811b-4efe-86a8-c4b586b38260","Type":"ContainerStarted","Data":"9bba0e4b69f0435a764e9c3d1c1cb247f80abdac03b0ca086fc5db3f55665a8c"} Jul 19 07:52:14.607366 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:52:14.607360 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-568d44cffc-nf2t8" Jul 19 07:52:14.607627 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:52:14.607376 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c4f6ccf57-kphwt" event={"ID":"f094d7c7-811b-4efe-86a8-c4b586b38260","Type":"ContainerStarted","Data":"fd785535c4a5a144bbd3b839a35d43ae1b9c12a9f30d1260b636eefc5f7cc09a"} Jul 19 07:52:14.620990 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:52:14.620947 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-rf2r2" podStartSLOduration=2.16663937 podStartE2EDuration="2.620929896s" totalImagesPullingTime="454.290526ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 07:52:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:52:14.618778601 +0000 UTC m=+600.973088202" watchObservedRunningTime="2026-07-19 07:52:14.620929896 +0000 UTC m=+600.975239497" Jul 19 07:52:14.640265 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:52:14.640238 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-568d44cffc-nf2t8"] Jul 19 07:52:14.645569 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:52:14.645539 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-568d44cffc-nf2t8"] Jul 19 07:52:14.657510 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:52:14.657465 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-c4f6ccf57-kphwt" podStartSLOduration=1.341676098 podStartE2EDuration="1.657449437s" totalImagesPullingTime="315.773339ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 07:52:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:52:14.65639415 +0000 UTC m=+601.010703753" watchObservedRunningTime="2026-07-19 07:52:14.657449437 +0000 UTC m=+601.011759037" Jul 19 07:52:14.684389 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:52:14.684360 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-2mjmd"] Jul 19 07:52:14.684606 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:52:14.684576 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-2mjmd" podUID="615e2145-0f67-436d-8dfa-50049bec3059" containerName="authorino" containerID="cri-o://e4db6fbba246c9194d0290745c73c1aca101c10cdd2bdc0898fb42e080211d9b" gracePeriod=30 Jul 19 07:52:14.961862 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:52:14.961839 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-rf2r2" Jul 19 07:52:14.965322 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:52:14.965302 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-2mjmd" Jul 19 07:52:15.086902 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:52:15.086800 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qb4qj\" (UniqueName: \"kubernetes.io/projected/bd8008ce-62d2-4372-a3a5-e64261c2be59-kube-api-access-qb4qj\") pod \"bd8008ce-62d2-4372-a3a5-e64261c2be59\" (UID: \"bd8008ce-62d2-4372-a3a5-e64261c2be59\") " Jul 19 07:52:15.086902 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:52:15.086861 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dkl8l\" (UniqueName: \"kubernetes.io/projected/615e2145-0f67-436d-8dfa-50049bec3059-kube-api-access-dkl8l\") pod \"615e2145-0f67-436d-8dfa-50049bec3059\" (UID: \"615e2145-0f67-436d-8dfa-50049bec3059\") " Jul 19 07:52:15.089056 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:52:15.089011 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/615e2145-0f67-436d-8dfa-50049bec3059-kube-api-access-dkl8l" (OuterVolumeSpecName: "kube-api-access-dkl8l") pod "615e2145-0f67-436d-8dfa-50049bec3059" (UID: "615e2145-0f67-436d-8dfa-50049bec3059"). InnerVolumeSpecName "kube-api-access-dkl8l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 07:52:15.089056 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:52:15.089026 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd8008ce-62d2-4372-a3a5-e64261c2be59-kube-api-access-qb4qj" (OuterVolumeSpecName: "kube-api-access-qb4qj") pod "bd8008ce-62d2-4372-a3a5-e64261c2be59" (UID: "bd8008ce-62d2-4372-a3a5-e64261c2be59"). InnerVolumeSpecName "kube-api-access-qb4qj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 07:52:15.188332 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:52:15.188291 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qb4qj\" (UniqueName: \"kubernetes.io/projected/bd8008ce-62d2-4372-a3a5-e64261c2be59-kube-api-access-qb4qj\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 07:52:15.188332 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:52:15.188322 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkl8l\" (UniqueName: \"kubernetes.io/projected/615e2145-0f67-436d-8dfa-50049bec3059-kube-api-access-dkl8l\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 07:52:15.613546 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:52:15.613489 2574 generic.go:356] "Generic (PLEG): container finished" podID="bd8008ce-62d2-4372-a3a5-e64261c2be59" containerID="00fba717a91869a738d496da86a8fdb4b4c2187dc8ba2dfd3f71c6f3871ed528" exitCode=0 Jul 19 07:52:15.613546 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:52:15.613552 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-rf2r2" Jul 19 07:52:15.614015 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:52:15.613560 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-rf2r2" event={"ID":"bd8008ce-62d2-4372-a3a5-e64261c2be59","Type":"ContainerDied","Data":"00fba717a91869a738d496da86a8fdb4b4c2187dc8ba2dfd3f71c6f3871ed528"} Jul 19 07:52:15.614015 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:52:15.613611 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-rf2r2" event={"ID":"bd8008ce-62d2-4372-a3a5-e64261c2be59","Type":"ContainerDied","Data":"7562f72ac703964c672019c521baef0e6661344a35b9323a583daee2bb2e8cac"} Jul 19 07:52:15.614015 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:52:15.613631 2574 scope.go:117] "RemoveContainer" containerID="00fba717a91869a738d496da86a8fdb4b4c2187dc8ba2dfd3f71c6f3871ed528" Jul 19 07:52:15.615179 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:52:15.615148 2574 generic.go:356] "Generic (PLEG): container finished" podID="615e2145-0f67-436d-8dfa-50049bec3059" containerID="e4db6fbba246c9194d0290745c73c1aca101c10cdd2bdc0898fb42e080211d9b" exitCode=0 Jul 19 07:52:15.615270 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:52:15.615185 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-2mjmd" Jul 19 07:52:15.615270 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:52:15.615224 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-2mjmd" event={"ID":"615e2145-0f67-436d-8dfa-50049bec3059","Type":"ContainerDied","Data":"e4db6fbba246c9194d0290745c73c1aca101c10cdd2bdc0898fb42e080211d9b"} Jul 19 07:52:15.615270 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:52:15.615255 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-2mjmd" event={"ID":"615e2145-0f67-436d-8dfa-50049bec3059","Type":"ContainerDied","Data":"55228ff8d102d7f0a274fa757bd1d620fbd08253ad9099f5713f2359ab81d7a2"} Jul 19 07:52:15.624257 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:52:15.624243 2574 scope.go:117] "RemoveContainer" containerID="00fba717a91869a738d496da86a8fdb4b4c2187dc8ba2dfd3f71c6f3871ed528" Jul 19 07:52:15.624600 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:52:15.624572 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00fba717a91869a738d496da86a8fdb4b4c2187dc8ba2dfd3f71c6f3871ed528\": container with ID starting with 00fba717a91869a738d496da86a8fdb4b4c2187dc8ba2dfd3f71c6f3871ed528 not found: ID does not exist" containerID="00fba717a91869a738d496da86a8fdb4b4c2187dc8ba2dfd3f71c6f3871ed528" Jul 19 07:52:15.624714 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:52:15.624612 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00fba717a91869a738d496da86a8fdb4b4c2187dc8ba2dfd3f71c6f3871ed528"} err="failed to get container status \"00fba717a91869a738d496da86a8fdb4b4c2187dc8ba2dfd3f71c6f3871ed528\": rpc error: code = NotFound desc = could not find container \"00fba717a91869a738d496da86a8fdb4b4c2187dc8ba2dfd3f71c6f3871ed528\": container with ID starting with 00fba717a91869a738d496da86a8fdb4b4c2187dc8ba2dfd3f71c6f3871ed528 not found: ID does not exist" Jul 19 07:52:15.624714 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:52:15.624638 2574 scope.go:117] "RemoveContainer" containerID="e4db6fbba246c9194d0290745c73c1aca101c10cdd2bdc0898fb42e080211d9b" Jul 19 07:52:15.632667 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:52:15.632610 2574 scope.go:117] "RemoveContainer" containerID="e4db6fbba246c9194d0290745c73c1aca101c10cdd2bdc0898fb42e080211d9b" Jul 19 07:52:15.632972 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:52:15.632945 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4db6fbba246c9194d0290745c73c1aca101c10cdd2bdc0898fb42e080211d9b\": container with ID starting with e4db6fbba246c9194d0290745c73c1aca101c10cdd2bdc0898fb42e080211d9b not found: ID does not exist" containerID="e4db6fbba246c9194d0290745c73c1aca101c10cdd2bdc0898fb42e080211d9b" Jul 19 07:52:15.633102 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:52:15.632979 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4db6fbba246c9194d0290745c73c1aca101c10cdd2bdc0898fb42e080211d9b"} err="failed to get container status \"e4db6fbba246c9194d0290745c73c1aca101c10cdd2bdc0898fb42e080211d9b\": rpc error: code = NotFound desc = could not find container \"e4db6fbba246c9194d0290745c73c1aca101c10cdd2bdc0898fb42e080211d9b\": container with ID starting with e4db6fbba246c9194d0290745c73c1aca101c10cdd2bdc0898fb42e080211d9b not found: ID does not exist" Jul 19 07:52:15.634874 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:52:15.634855 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-rf2r2"] Jul 19 07:52:15.638580 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:52:15.638557 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-rf2r2"] Jul 19 07:52:15.648092 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:52:15.648073 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-2mjmd"] Jul 19 07:52:15.649862 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:52:15.649844 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-2mjmd"] Jul 19 07:52:16.558737 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:52:16.558704 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="032f007f-b7c8-4baf-8c6e-0305196413fd" path="/var/lib/kubelet/pods/032f007f-b7c8-4baf-8c6e-0305196413fd/volumes" Jul 19 07:52:16.558987 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:52:16.558974 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="615e2145-0f67-436d-8dfa-50049bec3059" path="/var/lib/kubelet/pods/615e2145-0f67-436d-8dfa-50049bec3059/volumes" Jul 19 07:52:16.559305 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:52:16.559292 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd8008ce-62d2-4372-a3a5-e64261c2be59" path="/var/lib/kubelet/pods/bd8008ce-62d2-4372-a3a5-e64261c2be59/volumes" Jul 19 07:52:19.264342 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:52:19.264303 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:52:19.353908 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:52:19.353868 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:52:19.436147 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:52:19.436082 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:52:19.498775 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:52:19.498740 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:52:22.868098 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:52:22.868053 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:52:22.966734 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:52:22.966688 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:52:42.817310 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:52:42.817228 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:52:44.083786 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:52:44.083740 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:52:54.567233 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:52:54.567096 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:52:54.660199 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:52:54.660159 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:52:55.175962 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:52:55.175923 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:52:55.334017 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:52:55.333979 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:52:55.710737 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:52:55.710684 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:52:55.814411 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:52:55.814366 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:52:56.052987 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:52:56.052897 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 07:52:56.300261 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:52:56.300228 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:52:56.355169 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:52:56.355076 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:53:04.396294 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:53:04.396241 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:53:04.463424 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:53:04.463380 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:53:05.612875 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:53:05.612830 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:53:25.314471 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:53:25.314433 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:53:25.866937 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:53:25.866894 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 07:54:03.899916 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:54:03.899879 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:54:04.771818 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:54:04.771780 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 07:54:13.674143 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:54:13.674055 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:54:14.884461 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:54:14.884421 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 07:54:26.277210 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:54:26.277168 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:54:27.069343 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:54:27.069291 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 07:54:34.506767 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:54:34.506730 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:54:36.774251 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:54:36.774212 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 07:54:38.502629 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:54:38.502580 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:54:46.272366 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:54:46.272334 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 07:54:57.393729 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:54:57.393684 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:54:57.492068 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:54:57.492036 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:56:11.715795 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:56:11.715753 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-54b5cb644d-775fd"] Jul 19 07:56:11.716257 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:56:11.716168 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd8008ce-62d2-4372-a3a5-e64261c2be59" containerName="authorino" Jul 19 07:56:11.716257 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:56:11.716181 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd8008ce-62d2-4372-a3a5-e64261c2be59" containerName="authorino" Jul 19 07:56:11.716257 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:56:11.716199 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="615e2145-0f67-436d-8dfa-50049bec3059" containerName="authorino" Jul 19 07:56:11.716257 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:56:11.716205 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="615e2145-0f67-436d-8dfa-50049bec3059" containerName="authorino" Jul 19 07:56:11.716402 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:56:11.716273 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="615e2145-0f67-436d-8dfa-50049bec3059" containerName="authorino" Jul 19 07:56:11.716402 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:56:11.716283 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd8008ce-62d2-4372-a3a5-e64261c2be59" containerName="authorino" Jul 19 07:56:11.719330 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:56:11.719308 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-54b5cb644d-775fd" Jul 19 07:56:11.728448 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:56:11.728389 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-54b5cb644d-775fd"] Jul 19 07:56:11.748745 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:56:11.748717 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7e666b4f-2534-4d09-889e-a23a63ecef62-tls-cert\") pod \"authorino-54b5cb644d-775fd\" (UID: \"7e666b4f-2534-4d09-889e-a23a63ecef62\") " pod="kuadrant-system/authorino-54b5cb644d-775fd" Jul 19 07:56:11.748904 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:56:11.748768 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8npbb\" (UniqueName: \"kubernetes.io/projected/7e666b4f-2534-4d09-889e-a23a63ecef62-kube-api-access-8npbb\") pod \"authorino-54b5cb644d-775fd\" (UID: \"7e666b4f-2534-4d09-889e-a23a63ecef62\") " pod="kuadrant-system/authorino-54b5cb644d-775fd" Jul 19 07:56:11.849541 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:56:11.849476 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8npbb\" (UniqueName: \"kubernetes.io/projected/7e666b4f-2534-4d09-889e-a23a63ecef62-kube-api-access-8npbb\") pod \"authorino-54b5cb644d-775fd\" (UID: \"7e666b4f-2534-4d09-889e-a23a63ecef62\") " pod="kuadrant-system/authorino-54b5cb644d-775fd" Jul 19 07:56:11.849727 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:56:11.849586 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7e666b4f-2534-4d09-889e-a23a63ecef62-tls-cert\") pod \"authorino-54b5cb644d-775fd\" (UID: \"7e666b4f-2534-4d09-889e-a23a63ecef62\") " pod="kuadrant-system/authorino-54b5cb644d-775fd" Jul 19 07:56:11.852020 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:56:11.851991 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7e666b4f-2534-4d09-889e-a23a63ecef62-tls-cert\") pod \"authorino-54b5cb644d-775fd\" (UID: \"7e666b4f-2534-4d09-889e-a23a63ecef62\") " pod="kuadrant-system/authorino-54b5cb644d-775fd" Jul 19 07:56:11.856468 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:56:11.856446 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8npbb\" (UniqueName: \"kubernetes.io/projected/7e666b4f-2534-4d09-889e-a23a63ecef62-kube-api-access-8npbb\") pod \"authorino-54b5cb644d-775fd\" (UID: \"7e666b4f-2534-4d09-889e-a23a63ecef62\") " pod="kuadrant-system/authorino-54b5cb644d-775fd" Jul 19 07:56:12.031216 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:56:12.031134 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-54b5cb644d-775fd" Jul 19 07:56:12.171960 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:56:12.171933 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-54b5cb644d-775fd"] Jul 19 07:56:12.174136 ip-10-0-136-70 kubenswrapper[2574]: W0719 07:56:12.174103 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7e666b4f_2534_4d09_889e_a23a63ecef62.slice/crio-736c6d055d612f092a08b306aee5f326586aba2c85ce9005c4743f13d9530928 WatchSource:0}: Error finding container 736c6d055d612f092a08b306aee5f326586aba2c85ce9005c4743f13d9530928: Status 404 returned error can't find the container with id 736c6d055d612f092a08b306aee5f326586aba2c85ce9005c4743f13d9530928 Jul 19 07:56:12.175675 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:56:12.175653 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 19 07:56:12.784288 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:56:12.784256 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-54b5cb644d-775fd" event={"ID":"7e666b4f-2534-4d09-889e-a23a63ecef62","Type":"ContainerStarted","Data":"bad33e288304509511712e0e65e9f4b5cc8d265f18ccbda55408f0a1c98848a8"} Jul 19 07:56:12.784815 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:56:12.784297 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-54b5cb644d-775fd" event={"ID":"7e666b4f-2534-4d09-889e-a23a63ecef62","Type":"ContainerStarted","Data":"736c6d055d612f092a08b306aee5f326586aba2c85ce9005c4743f13d9530928"} Jul 19 07:56:12.798646 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:56:12.798593 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-54b5cb644d-775fd" podStartSLOduration=1.285497343 podStartE2EDuration="1.798572132s" totalImagesPullingTime="513.074789ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 07:56:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:56:12.798111456 +0000 UTC m=+839.152421058" watchObservedRunningTime="2026-07-19 07:56:12.798572132 +0000 UTC m=+839.152881734" Jul 19 07:56:12.824234 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:56:12.824198 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c4f6ccf57-kphwt"] Jul 19 07:56:12.824973 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:56:12.824935 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-c4f6ccf57-kphwt" podUID="f094d7c7-811b-4efe-86a8-c4b586b38260" containerName="authorino" containerID="cri-o://9bba0e4b69f0435a764e9c3d1c1cb247f80abdac03b0ca086fc5db3f55665a8c" gracePeriod=30 Jul 19 07:56:13.098479 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:56:13.098450 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c4f6ccf57-kphwt" Jul 19 07:56:13.170645 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:56:13.170609 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nnjhb\" (UniqueName: \"kubernetes.io/projected/f094d7c7-811b-4efe-86a8-c4b586b38260-kube-api-access-nnjhb\") pod \"f094d7c7-811b-4efe-86a8-c4b586b38260\" (UID: \"f094d7c7-811b-4efe-86a8-c4b586b38260\") " Jul 19 07:56:13.170844 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:56:13.170719 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f094d7c7-811b-4efe-86a8-c4b586b38260-tls-cert\") pod \"f094d7c7-811b-4efe-86a8-c4b586b38260\" (UID: \"f094d7c7-811b-4efe-86a8-c4b586b38260\") " Jul 19 07:56:13.177542 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:56:13.177288 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f094d7c7-811b-4efe-86a8-c4b586b38260-kube-api-access-nnjhb" (OuterVolumeSpecName: "kube-api-access-nnjhb") pod "f094d7c7-811b-4efe-86a8-c4b586b38260" (UID: "f094d7c7-811b-4efe-86a8-c4b586b38260"). InnerVolumeSpecName "kube-api-access-nnjhb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 07:56:13.187454 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:56:13.187424 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f094d7c7-811b-4efe-86a8-c4b586b38260-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "f094d7c7-811b-4efe-86a8-c4b586b38260" (UID: "f094d7c7-811b-4efe-86a8-c4b586b38260"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 07:56:13.272107 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:56:13.272068 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnjhb\" (UniqueName: \"kubernetes.io/projected/f094d7c7-811b-4efe-86a8-c4b586b38260-kube-api-access-nnjhb\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 07:56:13.272107 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:56:13.272102 2574 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f094d7c7-811b-4efe-86a8-c4b586b38260-tls-cert\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 07:56:13.310105 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:56:13.310066 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:56:13.411020 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:56:13.410974 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:56:13.790630 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:56:13.790597 2574 generic.go:356] "Generic (PLEG): container finished" podID="f094d7c7-811b-4efe-86a8-c4b586b38260" containerID="9bba0e4b69f0435a764e9c3d1c1cb247f80abdac03b0ca086fc5db3f55665a8c" exitCode=0 Jul 19 07:56:13.791134 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:56:13.790660 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c4f6ccf57-kphwt" Jul 19 07:56:13.791134 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:56:13.790684 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c4f6ccf57-kphwt" event={"ID":"f094d7c7-811b-4efe-86a8-c4b586b38260","Type":"ContainerDied","Data":"9bba0e4b69f0435a764e9c3d1c1cb247f80abdac03b0ca086fc5db3f55665a8c"} Jul 19 07:56:13.791134 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:56:13.790729 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c4f6ccf57-kphwt" event={"ID":"f094d7c7-811b-4efe-86a8-c4b586b38260","Type":"ContainerDied","Data":"fd785535c4a5a144bbd3b839a35d43ae1b9c12a9f30d1260b636eefc5f7cc09a"} Jul 19 07:56:13.791134 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:56:13.790748 2574 scope.go:117] "RemoveContainer" containerID="9bba0e4b69f0435a764e9c3d1c1cb247f80abdac03b0ca086fc5db3f55665a8c" Jul 19 07:56:13.799573 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:56:13.799554 2574 scope.go:117] "RemoveContainer" containerID="9bba0e4b69f0435a764e9c3d1c1cb247f80abdac03b0ca086fc5db3f55665a8c" Jul 19 07:56:13.799841 ip-10-0-136-70 kubenswrapper[2574]: E0719 07:56:13.799823 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bba0e4b69f0435a764e9c3d1c1cb247f80abdac03b0ca086fc5db3f55665a8c\": container with ID starting with 9bba0e4b69f0435a764e9c3d1c1cb247f80abdac03b0ca086fc5db3f55665a8c not found: ID does not exist" containerID="9bba0e4b69f0435a764e9c3d1c1cb247f80abdac03b0ca086fc5db3f55665a8c" Jul 19 07:56:13.799895 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:56:13.799851 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bba0e4b69f0435a764e9c3d1c1cb247f80abdac03b0ca086fc5db3f55665a8c"} err="failed to get container status \"9bba0e4b69f0435a764e9c3d1c1cb247f80abdac03b0ca086fc5db3f55665a8c\": rpc error: code = NotFound desc = could not find container \"9bba0e4b69f0435a764e9c3d1c1cb247f80abdac03b0ca086fc5db3f55665a8c\": container with ID starting with 9bba0e4b69f0435a764e9c3d1c1cb247f80abdac03b0ca086fc5db3f55665a8c not found: ID does not exist" Jul 19 07:56:13.818296 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:56:13.818266 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c4f6ccf57-kphwt"] Jul 19 07:56:13.820944 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:56:13.820912 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-c4f6ccf57-kphwt"] Jul 19 07:56:14.399897 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:56:14.399844 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:56:14.505289 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:56:14.505254 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:56:14.559743 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:56:14.559708 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f094d7c7-811b-4efe-86a8-c4b586b38260" path="/var/lib/kubelet/pods/f094d7c7-811b-4efe-86a8-c4b586b38260/volumes" Jul 19 07:56:50.775920 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:56:50.775871 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 07:57:00.493454 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:57:00.493416 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:57:01.071950 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:57:01.071911 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 07:57:12.679168 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:57:12.679081 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 07:57:14.534546 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:57:14.534301 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wmhfd_dd68d31d-f763-433a-867b-65b12eabac76/console-operator/1.log" Jul 19 07:57:14.536716 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:57:14.536686 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wmhfd_dd68d31d-f763-433a-867b-65b12eabac76/console-operator/1.log" Jul 19 07:58:07.454754 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:58:07.454707 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:58:08.360937 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:58:08.360900 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:58:27.398565 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:58:27.398508 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:58:28.610989 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:58:28.610943 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:58:51.569943 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:58:51.569847 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 07:59:02.102973 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:59:02.102932 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:59:02.206019 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:59:02.205966 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:59:02.312783 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:59:02.312731 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:59:08.000243 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:59:08.000207 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:59:08.105982 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:59:08.105934 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:59:08.200417 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:59:08.200374 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:59:08.339895 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:59:08.339804 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:59:14.012646 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:59:14.012582 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:59:14.113830 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:59:14.113789 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:59:31.469504 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:59:31.469462 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 07:59:40.873477 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:59:40.873434 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 07:59:51.125486 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:59:51.125439 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 07:59:51.672083 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:59:51.672044 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 07:59:59.270210 ip-10-0-136-70 kubenswrapper[2574]: I0719 07:59:59.270169 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 08:00:00.161454 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:00:00.161409 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29740800-6qc4k"] Jul 19 08:00:00.162421 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:00:00.162393 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f094d7c7-811b-4efe-86a8-c4b586b38260" containerName="authorino" Jul 19 08:00:00.162421 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:00:00.162423 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="f094d7c7-811b-4efe-86a8-c4b586b38260" containerName="authorino" Jul 19 08:00:00.162633 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:00:00.162618 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="f094d7c7-811b-4efe-86a8-c4b586b38260" containerName="authorino" Jul 19 08:00:00.168844 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:00:00.168816 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29740800-6qc4k" Jul 19 08:00:00.172633 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:00:00.172607 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qbdd4\"" Jul 19 08:00:00.177283 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:00:00.177259 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29740800-6qc4k"] Jul 19 08:00:00.180450 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:00:00.180426 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29740800-6qc4k" Jul 19 08:00:00.325824 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:00:00.325763 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29740800-6qc4k"] Jul 19 08:00:00.328839 ip-10-0-136-70 kubenswrapper[2574]: W0719 08:00:00.328809 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa862c8e_ccc2_4863_8edd_0430e652a9cc.slice/crio-b6175b381aa13b286f8de3657702496559742d51c4655ba1fbd3ff4f907010d6 WatchSource:0}: Error finding container b6175b381aa13b286f8de3657702496559742d51c4655ba1fbd3ff4f907010d6: Status 404 returned error can't find the container with id b6175b381aa13b286f8de3657702496559742d51c4655ba1fbd3ff4f907010d6 Jul 19 08:00:00.916125 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:00:00.916091 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29740800-6qc4k" event={"ID":"fa862c8e-ccc2-4863-8edd-0430e652a9cc","Type":"ContainerStarted","Data":"b6175b381aa13b286f8de3657702496559742d51c4655ba1fbd3ff4f907010d6"} Jul 19 08:00:02.636976 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:00:02.636936 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:00:03.934597 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:00:03.934554 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29740800-6qc4k" event={"ID":"fa862c8e-ccc2-4863-8edd-0430e652a9cc","Type":"ContainerStarted","Data":"bffa4c48535c680c7a3eece9bf77ec728c41784cb040071c5c63f841f1797f69"} Jul 19 08:00:03.949279 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:00:03.949224 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29740800-6qc4k" podStartSLOduration=1.1890802360000001 podStartE2EDuration="3.949205562s" totalImagesPullingTime="2.760125326s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 08:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:00:03.94870433 +0000 UTC m=+1070.303013940" watchObservedRunningTime="2026-07-19 08:00:03.949205562 +0000 UTC m=+1070.303515163" Jul 19 08:00:04.941942 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:00:04.941906 2574 generic.go:356] "Generic (PLEG): container finished" podID="fa862c8e-ccc2-4863-8edd-0430e652a9cc" containerID="bffa4c48535c680c7a3eece9bf77ec728c41784cb040071c5c63f841f1797f69" exitCode=0 Jul 19 08:00:04.942370 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:00:04.941994 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29740800-6qc4k" event={"ID":"fa862c8e-ccc2-4863-8edd-0430e652a9cc","Type":"ContainerDied","Data":"bffa4c48535c680c7a3eece9bf77ec728c41784cb040071c5c63f841f1797f69"} Jul 19 08:00:06.097792 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:00:06.097768 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29740800-6qc4k" Jul 19 08:00:06.955412 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:00:06.955328 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29740800-6qc4k" Jul 19 08:00:06.955604 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:00:06.955350 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29740800-6qc4k" event={"ID":"fa862c8e-ccc2-4863-8edd-0430e652a9cc","Type":"ContainerDied","Data":"b6175b381aa13b286f8de3657702496559742d51c4655ba1fbd3ff4f907010d6"} Jul 19 08:00:06.955604 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:00:06.955491 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b6175b381aa13b286f8de3657702496559742d51c4655ba1fbd3ff4f907010d6" Jul 19 08:00:10.972221 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:00:10.972184 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 08:00:22.268934 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:00:22.268891 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 08:00:33.374709 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:00:33.374671 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 08:00:43.570055 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:00:43.570017 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 08:01:08.678631 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:01:08.678594 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 08:01:18.681478 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:01:18.681440 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 08:01:41.574025 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:01:41.573928 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 08:01:52.576592 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:01:52.576547 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 08:02:08.173840 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:02:08.173802 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 08:02:14.567428 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:02:14.567399 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wmhfd_dd68d31d-f763-433a-867b-65b12eabac76/console-operator/1.log" Jul 19 08:02:14.568940 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:02:14.568914 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wmhfd_dd68d31d-f763-433a-867b-65b12eabac76/console-operator/1.log" Jul 19 08:02:17.977841 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:02:17.977800 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 08:02:28.471545 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:02:28.471496 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 08:02:38.473139 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:02:38.473105 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 08:02:48.035593 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:02:48.035550 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 08:03:12.747512 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:03:12.747417 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-szh92"] Jul 19 08:03:12.750184 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:03:12.747657 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-szh92" podUID="438041f9-2e00-4d37-9d04-2ab07314e1dc" containerName="manager" containerID="cri-o://3ca6f976d1b43f233b02622c6f1a13a8eb610f7cf5e948500350db2405619373" gracePeriod=10 Jul 19 08:03:15.710359 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:03:15.710334 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-szh92" Jul 19 08:03:15.798421 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:03:15.798320 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4nstz\" (UniqueName: \"kubernetes.io/projected/438041f9-2e00-4d37-9d04-2ab07314e1dc-kube-api-access-4nstz\") pod \"438041f9-2e00-4d37-9d04-2ab07314e1dc\" (UID: \"438041f9-2e00-4d37-9d04-2ab07314e1dc\") " Jul 19 08:03:15.798421 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:03:15.798380 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/438041f9-2e00-4d37-9d04-2ab07314e1dc-extensions-socket-volume\") pod \"438041f9-2e00-4d37-9d04-2ab07314e1dc\" (UID: \"438041f9-2e00-4d37-9d04-2ab07314e1dc\") " Jul 19 08:03:15.798830 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:03:15.798800 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/438041f9-2e00-4d37-9d04-2ab07314e1dc-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "438041f9-2e00-4d37-9d04-2ab07314e1dc" (UID: "438041f9-2e00-4d37-9d04-2ab07314e1dc"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 08:03:15.800622 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:03:15.800588 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/438041f9-2e00-4d37-9d04-2ab07314e1dc-kube-api-access-4nstz" (OuterVolumeSpecName: "kube-api-access-4nstz") pod "438041f9-2e00-4d37-9d04-2ab07314e1dc" (UID: "438041f9-2e00-4d37-9d04-2ab07314e1dc"). InnerVolumeSpecName "kube-api-access-4nstz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 08:03:15.899611 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:03:15.899581 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nstz\" (UniqueName: \"kubernetes.io/projected/438041f9-2e00-4d37-9d04-2ab07314e1dc-kube-api-access-4nstz\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:03:15.899611 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:03:15.899608 2574 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/438041f9-2e00-4d37-9d04-2ab07314e1dc-extensions-socket-volume\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:03:15.904647 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:03:15.904614 2574 generic.go:356] "Generic (PLEG): container finished" podID="438041f9-2e00-4d37-9d04-2ab07314e1dc" containerID="3ca6f976d1b43f233b02622c6f1a13a8eb610f7cf5e948500350db2405619373" exitCode=0 Jul 19 08:03:15.904775 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:03:15.904664 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-szh92" Jul 19 08:03:15.904775 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:03:15.904677 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-szh92" event={"ID":"438041f9-2e00-4d37-9d04-2ab07314e1dc","Type":"ContainerDied","Data":"3ca6f976d1b43f233b02622c6f1a13a8eb610f7cf5e948500350db2405619373"} Jul 19 08:03:15.904775 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:03:15.904727 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-szh92" event={"ID":"438041f9-2e00-4d37-9d04-2ab07314e1dc","Type":"ContainerDied","Data":"ac17b124e11fd62d7f4003099aea8763d0dbd3ae72efec9cdaa18294e10a8fb8"} Jul 19 08:03:15.904775 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:03:15.904752 2574 scope.go:117] "RemoveContainer" containerID="3ca6f976d1b43f233b02622c6f1a13a8eb610f7cf5e948500350db2405619373" Jul 19 08:03:15.916899 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:03:15.916861 2574 scope.go:117] "RemoveContainer" containerID="3ca6f976d1b43f233b02622c6f1a13a8eb610f7cf5e948500350db2405619373" Jul 19 08:03:15.917195 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:03:15.917174 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ca6f976d1b43f233b02622c6f1a13a8eb610f7cf5e948500350db2405619373\": container with ID starting with 3ca6f976d1b43f233b02622c6f1a13a8eb610f7cf5e948500350db2405619373 not found: ID does not exist" containerID="3ca6f976d1b43f233b02622c6f1a13a8eb610f7cf5e948500350db2405619373" Jul 19 08:03:15.917261 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:03:15.917202 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ca6f976d1b43f233b02622c6f1a13a8eb610f7cf5e948500350db2405619373"} err="failed to get container status \"3ca6f976d1b43f233b02622c6f1a13a8eb610f7cf5e948500350db2405619373\": rpc error: code = NotFound desc = could not find container \"3ca6f976d1b43f233b02622c6f1a13a8eb610f7cf5e948500350db2405619373\": container with ID starting with 3ca6f976d1b43f233b02622c6f1a13a8eb610f7cf5e948500350db2405619373 not found: ID does not exist" Jul 19 08:03:15.934839 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:03:15.934814 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-szh92"] Jul 19 08:03:15.937936 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:03:15.937916 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-szh92"] Jul 19 08:03:16.560391 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:03:16.560356 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="438041f9-2e00-4d37-9d04-2ab07314e1dc" path="/var/lib/kubelet/pods/438041f9-2e00-4d37-9d04-2ab07314e1dc/volumes" Jul 19 08:04:13.833256 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:04:13.833216 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z4jm9"] Jul 19 08:04:13.833782 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:04:13.833628 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="438041f9-2e00-4d37-9d04-2ab07314e1dc" containerName="manager" Jul 19 08:04:13.833782 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:04:13.833642 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="438041f9-2e00-4d37-9d04-2ab07314e1dc" containerName="manager" Jul 19 08:04:13.833782 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:04:13.833680 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa862c8e-ccc2-4863-8edd-0430e652a9cc" containerName="cleanup" Jul 19 08:04:13.833782 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:04:13.833688 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa862c8e-ccc2-4863-8edd-0430e652a9cc" containerName="cleanup" Jul 19 08:04:13.833782 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:04:13.833764 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa862c8e-ccc2-4863-8edd-0430e652a9cc" containerName="cleanup" Jul 19 08:04:13.833782 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:04:13.833776 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="438041f9-2e00-4d37-9d04-2ab07314e1dc" containerName="manager" Jul 19 08:04:13.836839 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:04:13.836824 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z4jm9" Jul 19 08:04:13.839401 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:04:13.839379 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-zktnv\"" Jul 19 08:04:13.849478 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:04:13.849451 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z4jm9"] Jul 19 08:04:13.987806 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:04:13.987759 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2d62t\" (UniqueName: \"kubernetes.io/projected/80be3ae2-48da-4746-91f7-a32946f64b04-kube-api-access-2d62t\") pod \"kuadrant-operator-controller-manager-86878c769f-z4jm9\" (UID: \"80be3ae2-48da-4746-91f7-a32946f64b04\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z4jm9" Jul 19 08:04:13.988046 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:04:13.987823 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/80be3ae2-48da-4746-91f7-a32946f64b04-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-z4jm9\" (UID: \"80be3ae2-48da-4746-91f7-a32946f64b04\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z4jm9" Jul 19 08:04:14.089420 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:04:14.089309 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2d62t\" (UniqueName: \"kubernetes.io/projected/80be3ae2-48da-4746-91f7-a32946f64b04-kube-api-access-2d62t\") pod \"kuadrant-operator-controller-manager-86878c769f-z4jm9\" (UID: \"80be3ae2-48da-4746-91f7-a32946f64b04\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z4jm9" Jul 19 08:04:14.089420 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:04:14.089383 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/80be3ae2-48da-4746-91f7-a32946f64b04-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-z4jm9\" (UID: \"80be3ae2-48da-4746-91f7-a32946f64b04\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z4jm9" Jul 19 08:04:14.089801 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:04:14.089781 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/80be3ae2-48da-4746-91f7-a32946f64b04-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-z4jm9\" (UID: \"80be3ae2-48da-4746-91f7-a32946f64b04\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z4jm9" Jul 19 08:04:14.097942 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:04:14.097906 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2d62t\" (UniqueName: \"kubernetes.io/projected/80be3ae2-48da-4746-91f7-a32946f64b04-kube-api-access-2d62t\") pod \"kuadrant-operator-controller-manager-86878c769f-z4jm9\" (UID: \"80be3ae2-48da-4746-91f7-a32946f64b04\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z4jm9" Jul 19 08:04:14.147474 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:04:14.147434 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z4jm9" Jul 19 08:04:14.284695 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:04:14.284672 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z4jm9"] Jul 19 08:04:14.286899 ip-10-0-136-70 kubenswrapper[2574]: W0719 08:04:14.286873 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80be3ae2_48da_4746_91f7_a32946f64b04.slice/crio-adc711361edc86240e70194ae6bfd57ed1c68560671ffe22c5d8481df4e25cbc WatchSource:0}: Error finding container adc711361edc86240e70194ae6bfd57ed1c68560671ffe22c5d8481df4e25cbc: Status 404 returned error can't find the container with id adc711361edc86240e70194ae6bfd57ed1c68560671ffe22c5d8481df4e25cbc Jul 19 08:04:15.202970 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:04:15.202935 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z4jm9" event={"ID":"80be3ae2-48da-4746-91f7-a32946f64b04","Type":"ContainerStarted","Data":"76b65b1cbb00857e10266ce2b535ae5b417e67204bcaff3d5db08beb43320b2e"} Jul 19 08:04:15.203455 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:04:15.203144 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z4jm9" Jul 19 08:04:15.203455 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:04:15.203192 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z4jm9" event={"ID":"80be3ae2-48da-4746-91f7-a32946f64b04","Type":"ContainerStarted","Data":"adc711361edc86240e70194ae6bfd57ed1c68560671ffe22c5d8481df4e25cbc"} Jul 19 08:04:15.227843 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:04:15.227796 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z4jm9" podStartSLOduration=2.227782012 podStartE2EDuration="2.227782012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 08:04:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:04:15.224853567 +0000 UTC m=+1321.579163170" watchObservedRunningTime="2026-07-19 08:04:15.227782012 +0000 UTC m=+1321.582091612" Jul 19 08:04:26.209747 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:04:26.209705 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z4jm9" Jul 19 08:04:35.170003 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:04:35.169961 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 08:04:47.670182 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:04:47.670102 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 08:05:09.873003 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:05:09.872960 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 08:05:22.378552 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:05:22.378491 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 08:05:32.382088 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:05:32.382050 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 08:05:43.367176 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:05:43.367142 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 08:05:53.081016 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:05:53.080963 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 08:06:02.692486 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:06:02.692448 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 08:06:14.872959 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:06:14.872865 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 08:06:26.075629 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:06:26.075588 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 08:06:45.668066 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:06:45.668019 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 08:06:53.278270 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:06:53.278228 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 08:07:03.072239 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:07:03.072204 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 08:07:13.265513 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:07:13.265477 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 08:07:14.599952 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:07:14.599918 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wmhfd_dd68d31d-f763-433a-867b-65b12eabac76/console-operator/1.log" Jul 19 08:07:14.600826 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:07:14.600807 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wmhfd_dd68d31d-f763-433a-867b-65b12eabac76/console-operator/1.log" Jul 19 08:07:16.445742 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:07:16.445705 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:07:17.629468 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:07:17.629430 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:07:22.774965 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:07:22.774924 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 08:07:30.874159 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:07:30.874116 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 08:07:45.877090 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:07:45.877006 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 08:07:55.467672 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:07:55.467633 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 08:08:05.572794 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:08:05.572758 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 08:08:15.394585 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:08:15.394548 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 08:08:26.574734 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:08:26.574695 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 08:08:44.786278 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:08:44.786239 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 08:09:01.579786 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:09:01.579746 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 08:09:07.558623 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:09:07.558502 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:09:07.646061 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:09:07.646020 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:09:13.567164 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:09:13.567121 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 08:09:23.482850 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:09:23.482810 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 08:09:33.076767 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:09:33.076725 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 08:09:44.572310 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:09:44.572272 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 08:09:56.072834 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:09:56.072795 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 08:10:08.573169 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:10:08.573131 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 08:10:14.116776 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:10:14.116737 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:10:14.215314 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:10:14.215273 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:10:14.540894 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:10:14.540852 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:10:20.212354 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:10:20.212317 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:10:20.270139 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:10:20.270108 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:10:20.397968 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:10:20.397927 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:10:20.498594 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:10:20.498495 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:10:21.505213 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:10:21.505176 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:10:21.605399 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:10:21.605361 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:10:21.679396 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:10:21.679362 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ef124b28-969b9ccd7-g5sk7"] Jul 19 08:10:21.683407 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:10:21.683385 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef124b28-969b9ccd7-g5sk7" Jul 19 08:10:21.685812 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:10:21.685788 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-ef124b28-serving-cert\"" Jul 19 08:10:21.685957 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:10:21.685937 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-tdbkh\"" Jul 19 08:10:21.686042 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:10:21.686021 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 19 08:10:21.686864 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:10:21.686844 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 19 08:10:21.692738 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:10:21.692716 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ef124b28-969b9ccd7-g5sk7"] Jul 19 08:10:21.712829 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:10:21.712786 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:10:21.834661 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:10:21.834573 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0cf73e24-459f-470a-a9fd-c9b2af7a11b1-maas-api-tls\") pod \"maas-api-e2e-ait-ef124b28-969b9ccd7-g5sk7\" (UID: \"0cf73e24-459f-470a-a9fd-c9b2af7a11b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef124b28-969b9ccd7-g5sk7" Jul 19 08:10:21.834661 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:10:21.834626 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cf73e24-459f-470a-a9fd-c9b2af7a11b1-openshift-service-ca\") pod \"maas-api-e2e-ait-ef124b28-969b9ccd7-g5sk7\" (UID: \"0cf73e24-459f-470a-a9fd-c9b2af7a11b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef124b28-969b9ccd7-g5sk7" Jul 19 08:10:21.834869 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:10:21.834697 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tsc6\" (UniqueName: \"kubernetes.io/projected/0cf73e24-459f-470a-a9fd-c9b2af7a11b1-kube-api-access-9tsc6\") pod \"maas-api-e2e-ait-ef124b28-969b9ccd7-g5sk7\" (UID: \"0cf73e24-459f-470a-a9fd-c9b2af7a11b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef124b28-969b9ccd7-g5sk7" Jul 19 08:10:21.936052 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:10:21.936015 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0cf73e24-459f-470a-a9fd-c9b2af7a11b1-maas-api-tls\") pod \"maas-api-e2e-ait-ef124b28-969b9ccd7-g5sk7\" (UID: \"0cf73e24-459f-470a-a9fd-c9b2af7a11b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef124b28-969b9ccd7-g5sk7" Jul 19 08:10:21.936275 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:10:21.936075 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cf73e24-459f-470a-a9fd-c9b2af7a11b1-openshift-service-ca\") pod \"maas-api-e2e-ait-ef124b28-969b9ccd7-g5sk7\" (UID: \"0cf73e24-459f-470a-a9fd-c9b2af7a11b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef124b28-969b9ccd7-g5sk7" Jul 19 08:10:21.936275 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:10:21.936135 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tsc6\" (UniqueName: \"kubernetes.io/projected/0cf73e24-459f-470a-a9fd-c9b2af7a11b1-kube-api-access-9tsc6\") pod \"maas-api-e2e-ait-ef124b28-969b9ccd7-g5sk7\" (UID: \"0cf73e24-459f-470a-a9fd-c9b2af7a11b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef124b28-969b9ccd7-g5sk7" Jul 19 08:10:21.936275 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:10:21.936181 2574 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-ef124b28-serving-cert: secret "maas-api-e2e-ait-ef124b28-serving-cert" not found Jul 19 08:10:21.936275 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:10:21.936253 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0cf73e24-459f-470a-a9fd-c9b2af7a11b1-maas-api-tls podName:0cf73e24-459f-470a-a9fd-c9b2af7a11b1 nodeName:}" failed. No retries permitted until 2026-07-19 08:10:22.436237198 +0000 UTC m=+1688.790546778 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0cf73e24-459f-470a-a9fd-c9b2af7a11b1-maas-api-tls") pod "maas-api-e2e-ait-ef124b28-969b9ccd7-g5sk7" (UID: "0cf73e24-459f-470a-a9fd-c9b2af7a11b1") : secret "maas-api-e2e-ait-ef124b28-serving-cert" not found Jul 19 08:10:21.936874 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:10:21.936856 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cf73e24-459f-470a-a9fd-c9b2af7a11b1-openshift-service-ca\") pod \"maas-api-e2e-ait-ef124b28-969b9ccd7-g5sk7\" (UID: \"0cf73e24-459f-470a-a9fd-c9b2af7a11b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef124b28-969b9ccd7-g5sk7" Jul 19 08:10:21.944462 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:10:21.944440 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tsc6\" (UniqueName: \"kubernetes.io/projected/0cf73e24-459f-470a-a9fd-c9b2af7a11b1-kube-api-access-9tsc6\") pod \"maas-api-e2e-ait-ef124b28-969b9ccd7-g5sk7\" (UID: \"0cf73e24-459f-470a-a9fd-c9b2af7a11b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef124b28-969b9ccd7-g5sk7" Jul 19 08:10:22.025140 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:10:22.025099 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:10:22.187509 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:10:22.187414 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:10:22.287584 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:10:22.287547 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:10:22.442136 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:10:22.442039 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0cf73e24-459f-470a-a9fd-c9b2af7a11b1-maas-api-tls\") pod \"maas-api-e2e-ait-ef124b28-969b9ccd7-g5sk7\" (UID: \"0cf73e24-459f-470a-a9fd-c9b2af7a11b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef124b28-969b9ccd7-g5sk7" Jul 19 08:10:22.444605 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:10:22.444583 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0cf73e24-459f-470a-a9fd-c9b2af7a11b1-maas-api-tls\") pod \"maas-api-e2e-ait-ef124b28-969b9ccd7-g5sk7\" (UID: \"0cf73e24-459f-470a-a9fd-c9b2af7a11b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef124b28-969b9ccd7-g5sk7" Jul 19 08:10:22.595460 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:10:22.595423 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef124b28-969b9ccd7-g5sk7" Jul 19 08:10:22.736092 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:10:22.736064 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ef124b28-969b9ccd7-g5sk7"] Jul 19 08:10:22.738153 ip-10-0-136-70 kubenswrapper[2574]: W0719 08:10:22.738123 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cf73e24_459f_470a_a9fd_c9b2af7a11b1.slice/crio-a75f0a786e801abfd003c6aebb51e419ee9425aa36656fb3a4d493a8c40ef557 WatchSource:0}: Error finding container a75f0a786e801abfd003c6aebb51e419ee9425aa36656fb3a4d493a8c40ef557: Status 404 returned error can't find the container with id a75f0a786e801abfd003c6aebb51e419ee9425aa36656fb3a4d493a8c40ef557 Jul 19 08:10:22.739457 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:10:22.739434 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 19 08:10:23.068229 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:10:23.068137 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef124b28-969b9ccd7-g5sk7" event={"ID":"0cf73e24-459f-470a-a9fd-c9b2af7a11b1","Type":"ContainerStarted","Data":"a75f0a786e801abfd003c6aebb51e419ee9425aa36656fb3a4d493a8c40ef557"} Jul 19 08:10:23.919206 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:10:23.919114 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:10:24.676959 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:10:24.676923 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 08:10:26.086391 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:10:26.086358 2574 generic.go:356] "Generic (PLEG): container finished" podID="0cf73e24-459f-470a-a9fd-c9b2af7a11b1" containerID="23e2bddf9607b92adb3bf7389ff3fc83f18edac5d96966aeaac43dfb2e2ce405" exitCode=1 Jul 19 08:10:26.086830 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:10:26.086406 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef124b28-969b9ccd7-g5sk7" event={"ID":"0cf73e24-459f-470a-a9fd-c9b2af7a11b1","Type":"ContainerDied","Data":"23e2bddf9607b92adb3bf7389ff3fc83f18edac5d96966aeaac43dfb2e2ce405"} Jul 19 08:10:26.086830 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:10:26.086661 2574 scope.go:117] "RemoveContainer" containerID="23e2bddf9607b92adb3bf7389ff3fc83f18edac5d96966aeaac43dfb2e2ce405" Jul 19 08:10:26.533979 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:10:26.533942 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-sz2jf"] Jul 19 08:10:26.537537 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:10:26.537499 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-sz2jf" Jul 19 08:10:26.539995 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:10:26.539975 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qbdd4\"" Jul 19 08:10:26.544791 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:10:26.544722 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-sz2jf"] Jul 19 08:10:26.680973 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:10:26.680889 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07ef403c-be0d-450d-82c2-32a749670405-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef124b28-sz2jf\" (UID: \"07ef403c-be0d-450d-82c2-32a749670405\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-sz2jf" Jul 19 08:10:26.782667 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:10:26.782631 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07ef403c-be0d-450d-82c2-32a749670405-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef124b28-sz2jf\" (UID: \"07ef403c-be0d-450d-82c2-32a749670405\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-sz2jf" Jul 19 08:10:26.783268 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:10:26.783248 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07ef403c-be0d-450d-82c2-32a749670405-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef124b28-sz2jf\" (UID: \"07ef403c-be0d-450d-82c2-32a749670405\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-sz2jf" Jul 19 08:10:26.849655 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:10:26.849620 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-sz2jf" Jul 19 08:10:26.991969 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:10:26.991946 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-sz2jf"] Jul 19 08:10:26.994866 ip-10-0-136-70 kubenswrapper[2574]: W0719 08:10:26.994823 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07ef403c_be0d_450d_82c2_32a749670405.slice/crio-736c7309a323ba4f9f3d5173fe05fd4ff35971530d9f6a513c946ab763fbade0 WatchSource:0}: Error finding container 736c7309a323ba4f9f3d5173fe05fd4ff35971530d9f6a513c946ab763fbade0: Status 404 returned error can't find the container with id 736c7309a323ba4f9f3d5173fe05fd4ff35971530d9f6a513c946ab763fbade0 Jul 19 08:10:27.093577 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:10:27.093539 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-sz2jf" event={"ID":"07ef403c-be0d-450d-82c2-32a749670405","Type":"ContainerStarted","Data":"736c7309a323ba4f9f3d5173fe05fd4ff35971530d9f6a513c946ab763fbade0"} Jul 19 08:10:27.095473 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:10:27.095443 2574 generic.go:356] "Generic (PLEG): container finished" podID="0cf73e24-459f-470a-a9fd-c9b2af7a11b1" containerID="1eaf5fe546e0dbc95b46ee30a94b5638f0ffaefe5f5c26a319cd00822afb457c" exitCode=1 Jul 19 08:10:27.095643 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:10:27.095545 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef124b28-969b9ccd7-g5sk7" event={"ID":"0cf73e24-459f-470a-a9fd-c9b2af7a11b1","Type":"ContainerDied","Data":"1eaf5fe546e0dbc95b46ee30a94b5638f0ffaefe5f5c26a319cd00822afb457c"} Jul 19 08:10:27.095643 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:10:27.095603 2574 scope.go:117] "RemoveContainer" containerID="23e2bddf9607b92adb3bf7389ff3fc83f18edac5d96966aeaac43dfb2e2ce405" Jul 19 08:10:27.095785 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:10:27.095770 2574 scope.go:117] "RemoveContainer" containerID="1eaf5fe546e0dbc95b46ee30a94b5638f0ffaefe5f5c26a319cd00822afb457c" Jul 19 08:10:27.096015 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:10:27.095996 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-ef124b28-969b9ccd7-g5sk7_odh-ai-gateway-infra(0cf73e24-459f-470a-a9fd-c9b2af7a11b1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef124b28-969b9ccd7-g5sk7" podUID="0cf73e24-459f-470a-a9fd-c9b2af7a11b1" Jul 19 08:10:27.595579 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:10:27.595482 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef124b28-969b9ccd7-g5sk7" Jul 19 08:10:28.101416 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:10:28.101379 2574 generic.go:356] "Generic (PLEG): container finished" podID="07ef403c-be0d-450d-82c2-32a749670405" containerID="5c58871705023a124c519ccf310cb43f5d0a946d8cb2f89381dfe7e25c844bf3" exitCode=7 Jul 19 08:10:28.101871 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:10:28.101466 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-sz2jf" event={"ID":"07ef403c-be0d-450d-82c2-32a749670405","Type":"ContainerDied","Data":"5c58871705023a124c519ccf310cb43f5d0a946d8cb2f89381dfe7e25c844bf3"} Jul 19 08:10:28.101871 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:10:28.101662 2574 scope.go:117] "RemoveContainer" containerID="5c58871705023a124c519ccf310cb43f5d0a946d8cb2f89381dfe7e25c844bf3" Jul 19 08:10:28.103449 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:10:28.103427 2574 scope.go:117] "RemoveContainer" containerID="1eaf5fe546e0dbc95b46ee30a94b5638f0ffaefe5f5c26a319cd00822afb457c" Jul 19 08:10:28.106536 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:10:28.104384 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-ef124b28-969b9ccd7-g5sk7_odh-ai-gateway-infra(0cf73e24-459f-470a-a9fd-c9b2af7a11b1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef124b28-969b9ccd7-g5sk7" podUID="0cf73e24-459f-470a-a9fd-c9b2af7a11b1" Jul 19 08:10:29.110014 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:10:29.109923 2574 generic.go:356] "Generic (PLEG): container finished" podID="07ef403c-be0d-450d-82c2-32a749670405" containerID="7e05aca1bf728f53d8f6dca76e968715f826cfb781eb5612c53fd90743a63ec4" exitCode=7 Jul 19 08:10:29.110434 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:10:29.110008 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-sz2jf" event={"ID":"07ef403c-be0d-450d-82c2-32a749670405","Type":"ContainerDied","Data":"7e05aca1bf728f53d8f6dca76e968715f826cfb781eb5612c53fd90743a63ec4"} Jul 19 08:10:29.110434 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:10:29.110083 2574 scope.go:117] "RemoveContainer" containerID="5c58871705023a124c519ccf310cb43f5d0a946d8cb2f89381dfe7e25c844bf3" Jul 19 08:10:29.110434 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:10:29.110229 2574 scope.go:117] "RemoveContainer" containerID="7e05aca1bf728f53d8f6dca76e968715f826cfb781eb5612c53fd90743a63ec4" Jul 19 08:10:29.110593 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:10:29.110450 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ef124b28-sz2jf_odh-ai-gateway-infra(07ef403c-be0d-450d-82c2-32a749670405)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-sz2jf" podUID="07ef403c-be0d-450d-82c2-32a749670405" Jul 19 08:10:29.110593 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:10:29.110470 2574 scope.go:117] "RemoveContainer" containerID="1eaf5fe546e0dbc95b46ee30a94b5638f0ffaefe5f5c26a319cd00822afb457c" Jul 19 08:10:29.110692 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:10:29.110638 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-ef124b28-969b9ccd7-g5sk7_odh-ai-gateway-infra(0cf73e24-459f-470a-a9fd-c9b2af7a11b1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef124b28-969b9ccd7-g5sk7" podUID="0cf73e24-459f-470a-a9fd-c9b2af7a11b1" Jul 19 08:10:30.117282 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:10:30.117248 2574 scope.go:117] "RemoveContainer" containerID="7e05aca1bf728f53d8f6dca76e968715f826cfb781eb5612c53fd90743a63ec4" Jul 19 08:10:30.117698 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:10:30.117503 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ef124b28-sz2jf_odh-ai-gateway-infra(07ef403c-be0d-450d-82c2-32a749670405)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-sz2jf" podUID="07ef403c-be0d-450d-82c2-32a749670405" Jul 19 08:10:32.595693 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:10:32.595658 2574 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef124b28-969b9ccd7-g5sk7" Jul 19 08:10:32.596070 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:10:32.596046 2574 scope.go:117] "RemoveContainer" containerID="1eaf5fe546e0dbc95b46ee30a94b5638f0ffaefe5f5c26a319cd00822afb457c" Jul 19 08:10:32.596232 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:10:32.596213 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-ef124b28-969b9ccd7-g5sk7_odh-ai-gateway-infra(0cf73e24-459f-470a-a9fd-c9b2af7a11b1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef124b28-969b9ccd7-g5sk7" podUID="0cf73e24-459f-470a-a9fd-c9b2af7a11b1" Jul 19 08:10:41.553488 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:10:41.553410 2574 scope.go:117] "RemoveContainer" containerID="7e05aca1bf728f53d8f6dca76e968715f826cfb781eb5612c53fd90743a63ec4" Jul 19 08:10:42.178376 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:10:42.178336 2574 generic.go:356] "Generic (PLEG): container finished" podID="07ef403c-be0d-450d-82c2-32a749670405" containerID="92c9c529174955711a23914c5b5bcf1518f62dc7d53b82351d2c412a04a2ec57" exitCode=7 Jul 19 08:10:42.178555 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:10:42.178412 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-sz2jf" event={"ID":"07ef403c-be0d-450d-82c2-32a749670405","Type":"ContainerDied","Data":"92c9c529174955711a23914c5b5bcf1518f62dc7d53b82351d2c412a04a2ec57"} Jul 19 08:10:42.178555 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:10:42.178499 2574 scope.go:117] "RemoveContainer" containerID="7e05aca1bf728f53d8f6dca76e968715f826cfb781eb5612c53fd90743a63ec4" Jul 19 08:10:42.178785 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:10:42.178768 2574 scope.go:117] "RemoveContainer" containerID="92c9c529174955711a23914c5b5bcf1518f62dc7d53b82351d2c412a04a2ec57" Jul 19 08:10:42.179015 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:10:42.178995 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ef124b28-sz2jf_odh-ai-gateway-infra(07ef403c-be0d-450d-82c2-32a749670405)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-sz2jf" podUID="07ef403c-be0d-450d-82c2-32a749670405" Jul 19 08:10:43.205958 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:10:43.205921 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-sz2jf"] Jul 19 08:10:43.353700 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:10:43.353675 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-sz2jf" Jul 19 08:10:43.421943 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:10:43.421907 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07ef403c-be0d-450d-82c2-32a749670405-maas-api-service-ca\") pod \"07ef403c-be0d-450d-82c2-32a749670405\" (UID: \"07ef403c-be0d-450d-82c2-32a749670405\") " Jul 19 08:10:43.422289 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:10:43.422268 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07ef403c-be0d-450d-82c2-32a749670405-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "07ef403c-be0d-450d-82c2-32a749670405" (UID: "07ef403c-be0d-450d-82c2-32a749670405"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 08:10:43.522862 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:10:43.522823 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07ef403c-be0d-450d-82c2-32a749670405-maas-api-service-ca\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:10:44.191239 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:10:44.191206 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-sz2jf" Jul 19 08:10:44.191418 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:10:44.191205 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-sz2jf" event={"ID":"07ef403c-be0d-450d-82c2-32a749670405","Type":"ContainerDied","Data":"736c7309a323ba4f9f3d5173fe05fd4ff35971530d9f6a513c946ab763fbade0"} Jul 19 08:10:44.191418 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:10:44.191341 2574 scope.go:117] "RemoveContainer" containerID="92c9c529174955711a23914c5b5bcf1518f62dc7d53b82351d2c412a04a2ec57" Jul 19 08:10:44.212885 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:10:44.212862 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-sz2jf"] Jul 19 08:10:44.218597 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:10:44.218573 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-sz2jf"] Jul 19 08:10:44.558538 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:10:44.558498 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07ef403c-be0d-450d-82c2-32a749670405" path="/var/lib/kubelet/pods/07ef403c-be0d-450d-82c2-32a749670405/volumes" Jul 19 08:10:45.553639 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:10:45.553606 2574 scope.go:117] "RemoveContainer" containerID="1eaf5fe546e0dbc95b46ee30a94b5638f0ffaefe5f5c26a319cd00822afb457c" Jul 19 08:10:46.204838 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:10:46.204744 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef124b28-969b9ccd7-g5sk7" event={"ID":"0cf73e24-459f-470a-a9fd-c9b2af7a11b1","Type":"ContainerStarted","Data":"e36c98779d719ddd119717b3e94c564fa4f42770951baa3ae2cfd61045d7692e"} Jul 19 08:10:46.205011 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:10:46.204964 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef124b28-969b9ccd7-g5sk7" Jul 19 08:10:46.221977 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:10:46.221922 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef124b28-969b9ccd7-g5sk7" podStartSLOduration=22.192939301 podStartE2EDuration="25.221904146s" totalImagesPullingTime="3.028964845s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 08:10:21 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:10:46.220169608 +0000 UTC m=+1712.574479223" watchObservedRunningTime="2026-07-19 08:10:46.221904146 +0000 UTC m=+1712.576213747" Jul 19 08:10:47.211192 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:10:47.211159 2574 generic.go:356] "Generic (PLEG): container finished" podID="0cf73e24-459f-470a-a9fd-c9b2af7a11b1" containerID="e36c98779d719ddd119717b3e94c564fa4f42770951baa3ae2cfd61045d7692e" exitCode=1 Jul 19 08:10:47.211699 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:10:47.211245 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef124b28-969b9ccd7-g5sk7" event={"ID":"0cf73e24-459f-470a-a9fd-c9b2af7a11b1","Type":"ContainerDied","Data":"e36c98779d719ddd119717b3e94c564fa4f42770951baa3ae2cfd61045d7692e"} Jul 19 08:10:47.211699 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:10:47.211311 2574 scope.go:117] "RemoveContainer" containerID="1eaf5fe546e0dbc95b46ee30a94b5638f0ffaefe5f5c26a319cd00822afb457c" Jul 19 08:10:47.211699 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:10:47.211500 2574 scope.go:117] "RemoveContainer" containerID="e36c98779d719ddd119717b3e94c564fa4f42770951baa3ae2cfd61045d7692e" Jul 19 08:10:47.211815 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:10:47.211748 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-ef124b28-969b9ccd7-g5sk7_odh-ai-gateway-infra(0cf73e24-459f-470a-a9fd-c9b2af7a11b1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef124b28-969b9ccd7-g5sk7" podUID="0cf73e24-459f-470a-a9fd-c9b2af7a11b1" Jul 19 08:10:48.218387 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:10:48.218353 2574 scope.go:117] "RemoveContainer" containerID="e36c98779d719ddd119717b3e94c564fa4f42770951baa3ae2cfd61045d7692e" Jul 19 08:10:48.218853 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:10:48.218613 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-ef124b28-969b9ccd7-g5sk7_odh-ai-gateway-infra(0cf73e24-459f-470a-a9fd-c9b2af7a11b1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef124b28-969b9ccd7-g5sk7" podUID="0cf73e24-459f-470a-a9fd-c9b2af7a11b1" Jul 19 08:10:52.595707 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:10:52.595672 2574 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef124b28-969b9ccd7-g5sk7" Jul 19 08:10:52.596110 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:10:52.596053 2574 scope.go:117] "RemoveContainer" containerID="e36c98779d719ddd119717b3e94c564fa4f42770951baa3ae2cfd61045d7692e" Jul 19 08:10:52.596235 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:10:52.596216 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-ef124b28-969b9ccd7-g5sk7_odh-ai-gateway-infra(0cf73e24-459f-470a-a9fd-c9b2af7a11b1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef124b28-969b9ccd7-g5sk7" podUID="0cf73e24-459f-470a-a9fd-c9b2af7a11b1" Jul 19 08:11:03.552783 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:11:03.552750 2574 scope.go:117] "RemoveContainer" containerID="e36c98779d719ddd119717b3e94c564fa4f42770951baa3ae2cfd61045d7692e" Jul 19 08:11:03.553188 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:11:03.552947 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-ef124b28-969b9ccd7-g5sk7_odh-ai-gateway-infra(0cf73e24-459f-470a-a9fd-c9b2af7a11b1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef124b28-969b9ccd7-g5sk7" podUID="0cf73e24-459f-470a-a9fd-c9b2af7a11b1" Jul 19 08:11:16.576272 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:11:16.576235 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-gp8gh"] Jul 19 08:11:16.576728 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:11:16.576625 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07ef403c-be0d-450d-82c2-32a749670405" containerName="revoke-keys" Jul 19 08:11:16.576728 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:11:16.576637 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="07ef403c-be0d-450d-82c2-32a749670405" containerName="revoke-keys" Jul 19 08:11:16.576728 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:11:16.576648 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07ef403c-be0d-450d-82c2-32a749670405" containerName="revoke-keys" Jul 19 08:11:16.576728 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:11:16.576654 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="07ef403c-be0d-450d-82c2-32a749670405" containerName="revoke-keys" Jul 19 08:11:16.576728 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:11:16.576723 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="07ef403c-be0d-450d-82c2-32a749670405" containerName="revoke-keys" Jul 19 08:11:16.576728 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:11:16.576732 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="07ef403c-be0d-450d-82c2-32a749670405" containerName="revoke-keys" Jul 19 08:11:16.576945 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:11:16.576739 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="07ef403c-be0d-450d-82c2-32a749670405" containerName="revoke-keys" Jul 19 08:11:16.579736 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:11:16.579721 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-gp8gh" Jul 19 08:11:16.582126 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:11:16.582097 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qbdd4\"" Jul 19 08:11:16.585171 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:11:16.585144 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-gp8gh"] Jul 19 08:11:16.686706 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:11:16.686670 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec784049-ce22-4115-bfc1-d58870001274-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef124b28-gp8gh\" (UID: \"ec784049-ce22-4115-bfc1-d58870001274\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-gp8gh" Jul 19 08:11:16.788023 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:11:16.787985 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec784049-ce22-4115-bfc1-d58870001274-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef124b28-gp8gh\" (UID: \"ec784049-ce22-4115-bfc1-d58870001274\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-gp8gh" Jul 19 08:11:16.788672 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:11:16.788653 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec784049-ce22-4115-bfc1-d58870001274-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef124b28-gp8gh\" (UID: \"ec784049-ce22-4115-bfc1-d58870001274\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-gp8gh" Jul 19 08:11:16.892811 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:11:16.892714 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-gp8gh" Jul 19 08:11:17.029475 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:11:17.029446 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-gp8gh"] Jul 19 08:11:17.031247 ip-10-0-136-70 kubenswrapper[2574]: W0719 08:11:17.031220 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec784049_ce22_4115_bfc1_d58870001274.slice/crio-72ed4555801ba9d4b3b01629707bd6a04fb8b5667b387e04d20f8bff9d71cfa2 WatchSource:0}: Error finding container 72ed4555801ba9d4b3b01629707bd6a04fb8b5667b387e04d20f8bff9d71cfa2: Status 404 returned error can't find the container with id 72ed4555801ba9d4b3b01629707bd6a04fb8b5667b387e04d20f8bff9d71cfa2 Jul 19 08:11:17.362792 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:11:17.362748 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-gp8gh" event={"ID":"ec784049-ce22-4115-bfc1-d58870001274","Type":"ContainerStarted","Data":"d33559293df0e87416827abfce06375be5e3d20a8168eb2c4977f720c62eb0f7"} Jul 19 08:11:17.362955 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:11:17.362813 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-gp8gh" event={"ID":"ec784049-ce22-4115-bfc1-d58870001274","Type":"ContainerStarted","Data":"72ed4555801ba9d4b3b01629707bd6a04fb8b5667b387e04d20f8bff9d71cfa2"} Jul 19 08:11:17.377294 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:11:17.377236 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-gp8gh" podStartSLOduration=1.377219444 podStartE2EDuration="1.377219444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 08:11:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:11:17.376182225 +0000 UTC m=+1743.730491825" watchObservedRunningTime="2026-07-19 08:11:17.377219444 +0000 UTC m=+1743.731529044" Jul 19 08:11:18.369360 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:11:18.369323 2574 generic.go:356] "Generic (PLEG): container finished" podID="ec784049-ce22-4115-bfc1-d58870001274" containerID="d33559293df0e87416827abfce06375be5e3d20a8168eb2c4977f720c62eb0f7" exitCode=7 Jul 19 08:11:18.369775 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:11:18.369394 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-gp8gh" event={"ID":"ec784049-ce22-4115-bfc1-d58870001274","Type":"ContainerDied","Data":"d33559293df0e87416827abfce06375be5e3d20a8168eb2c4977f720c62eb0f7"} Jul 19 08:11:18.369775 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:11:18.369593 2574 scope.go:117] "RemoveContainer" containerID="d33559293df0e87416827abfce06375be5e3d20a8168eb2c4977f720c62eb0f7" Jul 19 08:11:18.552766 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:11:18.552738 2574 scope.go:117] "RemoveContainer" containerID="e36c98779d719ddd119717b3e94c564fa4f42770951baa3ae2cfd61045d7692e" Jul 19 08:11:19.376459 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:11:19.376367 2574 generic.go:356] "Generic (PLEG): container finished" podID="0cf73e24-459f-470a-a9fd-c9b2af7a11b1" containerID="b346be2b8f148d3473468b1727be71224509941e1584b95fe9b341eb68d4884b" exitCode=1 Jul 19 08:11:19.376459 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:11:19.376443 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef124b28-969b9ccd7-g5sk7" event={"ID":"0cf73e24-459f-470a-a9fd-c9b2af7a11b1","Type":"ContainerDied","Data":"b346be2b8f148d3473468b1727be71224509941e1584b95fe9b341eb68d4884b"} Jul 19 08:11:19.376992 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:11:19.376506 2574 scope.go:117] "RemoveContainer" containerID="e36c98779d719ddd119717b3e94c564fa4f42770951baa3ae2cfd61045d7692e" Jul 19 08:11:19.376992 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:11:19.376843 2574 scope.go:117] "RemoveContainer" containerID="b346be2b8f148d3473468b1727be71224509941e1584b95fe9b341eb68d4884b" Jul 19 08:11:19.377085 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:11:19.377065 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-ef124b28-969b9ccd7-g5sk7_odh-ai-gateway-infra(0cf73e24-459f-470a-a9fd-c9b2af7a11b1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef124b28-969b9ccd7-g5sk7" podUID="0cf73e24-459f-470a-a9fd-c9b2af7a11b1" Jul 19 08:11:19.378573 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:11:19.378513 2574 generic.go:356] "Generic (PLEG): container finished" podID="ec784049-ce22-4115-bfc1-d58870001274" containerID="7612233ccf242150871ab8570e79efdebc5c558e7a4fc68e2f060aa47449bd6e" exitCode=7 Jul 19 08:11:19.378700 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:11:19.378614 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-gp8gh" event={"ID":"ec784049-ce22-4115-bfc1-d58870001274","Type":"ContainerDied","Data":"7612233ccf242150871ab8570e79efdebc5c558e7a4fc68e2f060aa47449bd6e"} Jul 19 08:11:19.378821 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:11:19.378808 2574 scope.go:117] "RemoveContainer" containerID="7612233ccf242150871ab8570e79efdebc5c558e7a4fc68e2f060aa47449bd6e" Jul 19 08:11:19.379002 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:11:19.378987 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ef124b28-gp8gh_odh-ai-gateway-infra(ec784049-ce22-4115-bfc1-d58870001274)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-gp8gh" podUID="ec784049-ce22-4115-bfc1-d58870001274" Jul 19 08:11:19.387968 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:11:19.387952 2574 scope.go:117] "RemoveContainer" containerID="d33559293df0e87416827abfce06375be5e3d20a8168eb2c4977f720c62eb0f7" Jul 19 08:11:20.387075 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:11:20.387049 2574 scope.go:117] "RemoveContainer" containerID="7612233ccf242150871ab8570e79efdebc5c558e7a4fc68e2f060aa47449bd6e" Jul 19 08:11:20.387469 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:11:20.387227 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ef124b28-gp8gh_odh-ai-gateway-infra(ec784049-ce22-4115-bfc1-d58870001274)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-gp8gh" podUID="ec784049-ce22-4115-bfc1-d58870001274" Jul 19 08:11:21.205973 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:11:21.205928 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef124b28-969b9ccd7-g5sk7" Jul 19 08:11:21.206401 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:11:21.206383 2574 scope.go:117] "RemoveContainer" containerID="b346be2b8f148d3473468b1727be71224509941e1584b95fe9b341eb68d4884b" Jul 19 08:11:21.206630 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:11:21.206608 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-ef124b28-969b9ccd7-g5sk7_odh-ai-gateway-infra(0cf73e24-459f-470a-a9fd-c9b2af7a11b1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef124b28-969b9ccd7-g5sk7" podUID="0cf73e24-459f-470a-a9fd-c9b2af7a11b1" Jul 19 08:11:22.596398 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:11:22.596361 2574 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef124b28-969b9ccd7-g5sk7" Jul 19 08:11:22.596852 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:11:22.596782 2574 scope.go:117] "RemoveContainer" containerID="b346be2b8f148d3473468b1727be71224509941e1584b95fe9b341eb68d4884b" Jul 19 08:11:22.596991 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:11:22.596970 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-ef124b28-969b9ccd7-g5sk7_odh-ai-gateway-infra(0cf73e24-459f-470a-a9fd-c9b2af7a11b1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef124b28-969b9ccd7-g5sk7" podUID="0cf73e24-459f-470a-a9fd-c9b2af7a11b1" Jul 19 08:11:33.553359 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:11:33.553331 2574 scope.go:117] "RemoveContainer" containerID="b346be2b8f148d3473468b1727be71224509941e1584b95fe9b341eb68d4884b" Jul 19 08:11:33.553848 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:11:33.553417 2574 scope.go:117] "RemoveContainer" containerID="7612233ccf242150871ab8570e79efdebc5c558e7a4fc68e2f060aa47449bd6e" Jul 19 08:11:33.553848 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:11:33.553544 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-ef124b28-969b9ccd7-g5sk7_odh-ai-gateway-infra(0cf73e24-459f-470a-a9fd-c9b2af7a11b1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef124b28-969b9ccd7-g5sk7" podUID="0cf73e24-459f-470a-a9fd-c9b2af7a11b1" Jul 19 08:11:34.464212 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:11:34.464183 2574 generic.go:356] "Generic (PLEG): container finished" podID="ec784049-ce22-4115-bfc1-d58870001274" containerID="1ba42620cfb0101b71de7d74871840576caa32671e4110725d8990dfd38e382c" exitCode=7 Jul 19 08:11:34.464383 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:11:34.464252 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-gp8gh" event={"ID":"ec784049-ce22-4115-bfc1-d58870001274","Type":"ContainerDied","Data":"1ba42620cfb0101b71de7d74871840576caa32671e4110725d8990dfd38e382c"} Jul 19 08:11:34.464383 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:11:34.464311 2574 scope.go:117] "RemoveContainer" containerID="7612233ccf242150871ab8570e79efdebc5c558e7a4fc68e2f060aa47449bd6e" Jul 19 08:11:34.464692 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:11:34.464642 2574 scope.go:117] "RemoveContainer" containerID="1ba42620cfb0101b71de7d74871840576caa32671e4110725d8990dfd38e382c" Jul 19 08:11:34.464860 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:11:34.464839 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ef124b28-gp8gh_odh-ai-gateway-infra(ec784049-ce22-4115-bfc1-d58870001274)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-gp8gh" podUID="ec784049-ce22-4115-bfc1-d58870001274" Jul 19 08:11:35.489673 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:11:35.489633 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-gp8gh"] Jul 19 08:11:35.630874 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:11:35.630846 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-gp8gh" Jul 19 08:11:35.757009 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:11:35.756899 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec784049-ce22-4115-bfc1-d58870001274-maas-api-service-ca\") pod \"ec784049-ce22-4115-bfc1-d58870001274\" (UID: \"ec784049-ce22-4115-bfc1-d58870001274\") " Jul 19 08:11:35.757275 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:11:35.757250 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec784049-ce22-4115-bfc1-d58870001274-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ec784049-ce22-4115-bfc1-d58870001274" (UID: "ec784049-ce22-4115-bfc1-d58870001274"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 08:11:35.858498 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:11:35.858461 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec784049-ce22-4115-bfc1-d58870001274-maas-api-service-ca\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:11:36.476347 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:11:36.476320 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-gp8gh" Jul 19 08:11:36.476555 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:11:36.476321 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-gp8gh" event={"ID":"ec784049-ce22-4115-bfc1-d58870001274","Type":"ContainerDied","Data":"72ed4555801ba9d4b3b01629707bd6a04fb8b5667b387e04d20f8bff9d71cfa2"} Jul 19 08:11:36.476555 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:11:36.476453 2574 scope.go:117] "RemoveContainer" containerID="1ba42620cfb0101b71de7d74871840576caa32671e4110725d8990dfd38e382c" Jul 19 08:11:36.496340 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:11:36.496315 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-gp8gh"] Jul 19 08:11:36.502342 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:11:36.502317 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-gp8gh"] Jul 19 08:11:36.558987 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:11:36.558944 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec784049-ce22-4115-bfc1-d58870001274" path="/var/lib/kubelet/pods/ec784049-ce22-4115-bfc1-d58870001274/volumes" Jul 19 08:11:44.554754 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:11:44.554723 2574 scope.go:117] "RemoveContainer" containerID="b346be2b8f148d3473468b1727be71224509941e1584b95fe9b341eb68d4884b" Jul 19 08:11:44.555172 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:11:44.554923 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-ef124b28-969b9ccd7-g5sk7_odh-ai-gateway-infra(0cf73e24-459f-470a-a9fd-c9b2af7a11b1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef124b28-969b9ccd7-g5sk7" podUID="0cf73e24-459f-470a-a9fd-c9b2af7a11b1" Jul 19 08:11:55.553347 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:11:55.553312 2574 scope.go:117] "RemoveContainer" containerID="b346be2b8f148d3473468b1727be71224509941e1584b95fe9b341eb68d4884b" Jul 19 08:11:55.553788 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:11:55.553503 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-ef124b28-969b9ccd7-g5sk7_odh-ai-gateway-infra(0cf73e24-459f-470a-a9fd-c9b2af7a11b1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef124b28-969b9ccd7-g5sk7" podUID="0cf73e24-459f-470a-a9fd-c9b2af7a11b1" Jul 19 08:12:06.620119 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:12:06.620036 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-vcl6v"] Jul 19 08:12:06.620615 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:12:06.620429 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec784049-ce22-4115-bfc1-d58870001274" containerName="revoke-keys" Jul 19 08:12:06.620615 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:12:06.620440 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec784049-ce22-4115-bfc1-d58870001274" containerName="revoke-keys" Jul 19 08:12:06.620615 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:12:06.620452 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec784049-ce22-4115-bfc1-d58870001274" containerName="revoke-keys" Jul 19 08:12:06.620615 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:12:06.620457 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec784049-ce22-4115-bfc1-d58870001274" containerName="revoke-keys" Jul 19 08:12:06.620615 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:12:06.620466 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07ef403c-be0d-450d-82c2-32a749670405" containerName="revoke-keys" Jul 19 08:12:06.620615 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:12:06.620472 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="07ef403c-be0d-450d-82c2-32a749670405" containerName="revoke-keys" Jul 19 08:12:06.620615 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:12:06.620554 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec784049-ce22-4115-bfc1-d58870001274" containerName="revoke-keys" Jul 19 08:12:06.620615 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:12:06.620565 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec784049-ce22-4115-bfc1-d58870001274" containerName="revoke-keys" Jul 19 08:12:06.620615 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:12:06.620586 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec784049-ce22-4115-bfc1-d58870001274" containerName="revoke-keys" Jul 19 08:12:06.623669 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:12:06.623652 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-vcl6v" Jul 19 08:12:06.625977 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:12:06.625955 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qbdd4\"" Jul 19 08:12:06.630007 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:12:06.629902 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-vcl6v"] Jul 19 08:12:06.727719 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:12:06.727683 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75400e5a-56e3-4e6a-a8f4-2042ba9d90d6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef124b28-vcl6v\" (UID: \"75400e5a-56e3-4e6a-a8f4-2042ba9d90d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-vcl6v" Jul 19 08:12:06.828654 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:12:06.828597 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75400e5a-56e3-4e6a-a8f4-2042ba9d90d6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef124b28-vcl6v\" (UID: \"75400e5a-56e3-4e6a-a8f4-2042ba9d90d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-vcl6v" Jul 19 08:12:06.829307 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:12:06.829279 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75400e5a-56e3-4e6a-a8f4-2042ba9d90d6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef124b28-vcl6v\" (UID: \"75400e5a-56e3-4e6a-a8f4-2042ba9d90d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-vcl6v" Jul 19 08:12:06.937455 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:12:06.937356 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-vcl6v" Jul 19 08:12:07.072835 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:12:07.072802 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-vcl6v"] Jul 19 08:12:07.076511 ip-10-0-136-70 kubenswrapper[2574]: W0719 08:12:07.076480 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75400e5a_56e3_4e6a_a8f4_2042ba9d90d6.slice/crio-8c15f22a7d2cd95c3b9d1e673cd73db15a3e0f63a4058fbe33bdde795481c735 WatchSource:0}: Error finding container 8c15f22a7d2cd95c3b9d1e673cd73db15a3e0f63a4058fbe33bdde795481c735: Status 404 returned error can't find the container with id 8c15f22a7d2cd95c3b9d1e673cd73db15a3e0f63a4058fbe33bdde795481c735 Jul 19 08:12:07.642229 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:12:07.642124 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-vcl6v" event={"ID":"75400e5a-56e3-4e6a-a8f4-2042ba9d90d6","Type":"ContainerStarted","Data":"12aaae5ab8e3d3e4b434d82b11e7472bb0a6ec6b3d43901c1045cb6c488da519"} Jul 19 08:12:07.642229 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:12:07.642186 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-vcl6v" event={"ID":"75400e5a-56e3-4e6a-a8f4-2042ba9d90d6","Type":"ContainerStarted","Data":"8c15f22a7d2cd95c3b9d1e673cd73db15a3e0f63a4058fbe33bdde795481c735"} Jul 19 08:12:07.656590 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:12:07.656536 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-vcl6v" podStartSLOduration=1.656504623 podStartE2EDuration="1.656504623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 08:12:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:12:07.655251852 +0000 UTC m=+1794.009561454" watchObservedRunningTime="2026-07-19 08:12:07.656504623 +0000 UTC m=+1794.010814213" Jul 19 08:12:08.648376 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:12:08.648336 2574 generic.go:356] "Generic (PLEG): container finished" podID="75400e5a-56e3-4e6a-a8f4-2042ba9d90d6" containerID="12aaae5ab8e3d3e4b434d82b11e7472bb0a6ec6b3d43901c1045cb6c488da519" exitCode=7 Jul 19 08:12:08.648815 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:12:08.648414 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-vcl6v" event={"ID":"75400e5a-56e3-4e6a-a8f4-2042ba9d90d6","Type":"ContainerDied","Data":"12aaae5ab8e3d3e4b434d82b11e7472bb0a6ec6b3d43901c1045cb6c488da519"} Jul 19 08:12:08.648815 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:12:08.648656 2574 scope.go:117] "RemoveContainer" containerID="12aaae5ab8e3d3e4b434d82b11e7472bb0a6ec6b3d43901c1045cb6c488da519" Jul 19 08:12:09.655551 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:12:09.655410 2574 generic.go:356] "Generic (PLEG): container finished" podID="75400e5a-56e3-4e6a-a8f4-2042ba9d90d6" containerID="6e30a12486a05b207e47d0e75890f28aa598d190430df7f608407bdc537c5a86" exitCode=7 Jul 19 08:12:09.655551 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:12:09.655500 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-vcl6v" event={"ID":"75400e5a-56e3-4e6a-a8f4-2042ba9d90d6","Type":"ContainerDied","Data":"6e30a12486a05b207e47d0e75890f28aa598d190430df7f608407bdc537c5a86"} Jul 19 08:12:09.656039 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:12:09.655609 2574 scope.go:117] "RemoveContainer" containerID="12aaae5ab8e3d3e4b434d82b11e7472bb0a6ec6b3d43901c1045cb6c488da519" Jul 19 08:12:09.656039 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:12:09.655705 2574 scope.go:117] "RemoveContainer" containerID="6e30a12486a05b207e47d0e75890f28aa598d190430df7f608407bdc537c5a86" Jul 19 08:12:09.656039 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:12:09.655928 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ef124b28-vcl6v_odh-ai-gateway-infra(75400e5a-56e3-4e6a-a8f4-2042ba9d90d6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-vcl6v" podUID="75400e5a-56e3-4e6a-a8f4-2042ba9d90d6" Jul 19 08:12:10.552665 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:12:10.552629 2574 scope.go:117] "RemoveContainer" containerID="b346be2b8f148d3473468b1727be71224509941e1584b95fe9b341eb68d4884b" Jul 19 08:12:10.662935 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:12:10.662905 2574 scope.go:117] "RemoveContainer" containerID="6e30a12486a05b207e47d0e75890f28aa598d190430df7f608407bdc537c5a86" Jul 19 08:12:10.663326 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:12:10.663083 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ef124b28-vcl6v_odh-ai-gateway-infra(75400e5a-56e3-4e6a-a8f4-2042ba9d90d6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-vcl6v" podUID="75400e5a-56e3-4e6a-a8f4-2042ba9d90d6" Jul 19 08:12:11.670503 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:12:11.670413 2574 generic.go:356] "Generic (PLEG): container finished" podID="0cf73e24-459f-470a-a9fd-c9b2af7a11b1" containerID="8d5c0e9e71b976ca7d6025339747a9f6d76e141dc863df70935949548a6a9065" exitCode=1 Jul 19 08:12:11.670503 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:12:11.670491 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef124b28-969b9ccd7-g5sk7" event={"ID":"0cf73e24-459f-470a-a9fd-c9b2af7a11b1","Type":"ContainerDied","Data":"8d5c0e9e71b976ca7d6025339747a9f6d76e141dc863df70935949548a6a9065"} Jul 19 08:12:11.670972 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:12:11.670579 2574 scope.go:117] "RemoveContainer" containerID="b346be2b8f148d3473468b1727be71224509941e1584b95fe9b341eb68d4884b" Jul 19 08:12:11.670972 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:12:11.670888 2574 scope.go:117] "RemoveContainer" containerID="8d5c0e9e71b976ca7d6025339747a9f6d76e141dc863df70935949548a6a9065" Jul 19 08:12:11.671086 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:12:11.671065 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-ef124b28-969b9ccd7-g5sk7_odh-ai-gateway-infra(0cf73e24-459f-470a-a9fd-c9b2af7a11b1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef124b28-969b9ccd7-g5sk7" podUID="0cf73e24-459f-470a-a9fd-c9b2af7a11b1" Jul 19 08:12:12.596261 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:12:12.596231 2574 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef124b28-969b9ccd7-g5sk7" Jul 19 08:12:12.681308 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:12:12.681278 2574 scope.go:117] "RemoveContainer" containerID="8d5c0e9e71b976ca7d6025339747a9f6d76e141dc863df70935949548a6a9065" Jul 19 08:12:12.681731 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:12:12.681449 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-ef124b28-969b9ccd7-g5sk7_odh-ai-gateway-infra(0cf73e24-459f-470a-a9fd-c9b2af7a11b1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef124b28-969b9ccd7-g5sk7" podUID="0cf73e24-459f-470a-a9fd-c9b2af7a11b1" Jul 19 08:12:14.629674 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:12:14.629639 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wmhfd_dd68d31d-f763-433a-867b-65b12eabac76/console-operator/1.log" Jul 19 08:12:14.631786 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:12:14.631760 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wmhfd_dd68d31d-f763-433a-867b-65b12eabac76/console-operator/1.log" Jul 19 08:12:16.205574 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:12:16.205534 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef124b28-969b9ccd7-g5sk7" Jul 19 08:12:16.205952 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:12:16.205920 2574 scope.go:117] "RemoveContainer" containerID="8d5c0e9e71b976ca7d6025339747a9f6d76e141dc863df70935949548a6a9065" Jul 19 08:12:16.206108 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:12:16.206085 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-ef124b28-969b9ccd7-g5sk7_odh-ai-gateway-infra(0cf73e24-459f-470a-a9fd-c9b2af7a11b1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef124b28-969b9ccd7-g5sk7" podUID="0cf73e24-459f-470a-a9fd-c9b2af7a11b1" Jul 19 08:12:24.555265 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:12:24.555231 2574 scope.go:117] "RemoveContainer" containerID="6e30a12486a05b207e47d0e75890f28aa598d190430df7f608407bdc537c5a86" Jul 19 08:12:25.754196 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:12:25.754158 2574 generic.go:356] "Generic (PLEG): container finished" podID="75400e5a-56e3-4e6a-a8f4-2042ba9d90d6" containerID="0ad8f7fc249919afd35b655fc8cfec1d695a789311bbb331ee143e63a489f4e9" exitCode=7 Jul 19 08:12:25.754688 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:12:25.754207 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-vcl6v" event={"ID":"75400e5a-56e3-4e6a-a8f4-2042ba9d90d6","Type":"ContainerDied","Data":"0ad8f7fc249919afd35b655fc8cfec1d695a789311bbb331ee143e63a489f4e9"} Jul 19 08:12:25.754688 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:12:25.754284 2574 scope.go:117] "RemoveContainer" containerID="6e30a12486a05b207e47d0e75890f28aa598d190430df7f608407bdc537c5a86" Jul 19 08:12:25.754688 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:12:25.754564 2574 scope.go:117] "RemoveContainer" containerID="0ad8f7fc249919afd35b655fc8cfec1d695a789311bbb331ee143e63a489f4e9" Jul 19 08:12:25.754861 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:12:25.754836 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ef124b28-vcl6v_odh-ai-gateway-infra(75400e5a-56e3-4e6a-a8f4-2042ba9d90d6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-vcl6v" podUID="75400e5a-56e3-4e6a-a8f4-2042ba9d90d6" Jul 19 08:12:26.777667 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:12:26.777632 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-vcl6v"] Jul 19 08:12:26.922118 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:12:26.922091 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-vcl6v" Jul 19 08:12:26.999256 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:12:26.999219 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75400e5a-56e3-4e6a-a8f4-2042ba9d90d6-maas-api-service-ca\") pod \"75400e5a-56e3-4e6a-a8f4-2042ba9d90d6\" (UID: \"75400e5a-56e3-4e6a-a8f4-2042ba9d90d6\") " Jul 19 08:12:26.999635 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:12:26.999612 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75400e5a-56e3-4e6a-a8f4-2042ba9d90d6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "75400e5a-56e3-4e6a-a8f4-2042ba9d90d6" (UID: "75400e5a-56e3-4e6a-a8f4-2042ba9d90d6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 08:12:27.100826 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:12:27.100738 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75400e5a-56e3-4e6a-a8f4-2042ba9d90d6-maas-api-service-ca\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:12:27.553653 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:12:27.553620 2574 scope.go:117] "RemoveContainer" containerID="8d5c0e9e71b976ca7d6025339747a9f6d76e141dc863df70935949548a6a9065" Jul 19 08:12:27.553841 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:12:27.553801 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-ef124b28-969b9ccd7-g5sk7_odh-ai-gateway-infra(0cf73e24-459f-470a-a9fd-c9b2af7a11b1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef124b28-969b9ccd7-g5sk7" podUID="0cf73e24-459f-470a-a9fd-c9b2af7a11b1" Jul 19 08:12:27.766897 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:12:27.766866 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-vcl6v" Jul 19 08:12:27.766897 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:12:27.766881 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-vcl6v" event={"ID":"75400e5a-56e3-4e6a-a8f4-2042ba9d90d6","Type":"ContainerDied","Data":"8c15f22a7d2cd95c3b9d1e673cd73db15a3e0f63a4058fbe33bdde795481c735"} Jul 19 08:12:27.767167 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:12:27.766942 2574 scope.go:117] "RemoveContainer" containerID="0ad8f7fc249919afd35b655fc8cfec1d695a789311bbb331ee143e63a489f4e9" Jul 19 08:12:27.789128 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:12:27.789097 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-vcl6v"] Jul 19 08:12:27.792568 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:12:27.792544 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-vcl6v"] Jul 19 08:12:28.559064 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:12:28.559026 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75400e5a-56e3-4e6a-a8f4-2042ba9d90d6" path="/var/lib/kubelet/pods/75400e5a-56e3-4e6a-a8f4-2042ba9d90d6/volumes" Jul 19 08:12:41.552876 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:12:41.552839 2574 scope.go:117] "RemoveContainer" containerID="8d5c0e9e71b976ca7d6025339747a9f6d76e141dc863df70935949548a6a9065" Jul 19 08:12:41.553473 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:12:41.553089 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-ef124b28-969b9ccd7-g5sk7_odh-ai-gateway-infra(0cf73e24-459f-470a-a9fd-c9b2af7a11b1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef124b28-969b9ccd7-g5sk7" podUID="0cf73e24-459f-470a-a9fd-c9b2af7a11b1" Jul 19 08:12:52.559029 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:12:52.558998 2574 scope.go:117] "RemoveContainer" containerID="8d5c0e9e71b976ca7d6025339747a9f6d76e141dc863df70935949548a6a9065" Jul 19 08:12:52.559439 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:12:52.559186 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-ef124b28-969b9ccd7-g5sk7_odh-ai-gateway-infra(0cf73e24-459f-470a-a9fd-c9b2af7a11b1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef124b28-969b9ccd7-g5sk7" podUID="0cf73e24-459f-470a-a9fd-c9b2af7a11b1" Jul 19 08:13:05.552942 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:05.552904 2574 scope.go:117] "RemoveContainer" containerID="8d5c0e9e71b976ca7d6025339747a9f6d76e141dc863df70935949548a6a9065" Jul 19 08:13:05.553377 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:13:05.553091 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-ef124b28-969b9ccd7-g5sk7_odh-ai-gateway-infra(0cf73e24-459f-470a-a9fd-c9b2af7a11b1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef124b28-969b9ccd7-g5sk7" podUID="0cf73e24-459f-470a-a9fd-c9b2af7a11b1" Jul 19 08:13:06.666340 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:06.666297 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-2hv7v"] Jul 19 08:13:06.666840 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:06.666820 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75400e5a-56e3-4e6a-a8f4-2042ba9d90d6" containerName="revoke-keys" Jul 19 08:13:06.666898 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:06.666844 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="75400e5a-56e3-4e6a-a8f4-2042ba9d90d6" containerName="revoke-keys" Jul 19 08:13:06.666898 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:06.666857 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75400e5a-56e3-4e6a-a8f4-2042ba9d90d6" containerName="revoke-keys" Jul 19 08:13:06.666898 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:06.666866 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="75400e5a-56e3-4e6a-a8f4-2042ba9d90d6" containerName="revoke-keys" Jul 19 08:13:06.667010 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:06.666899 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec784049-ce22-4115-bfc1-d58870001274" containerName="revoke-keys" Jul 19 08:13:06.667010 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:06.666907 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec784049-ce22-4115-bfc1-d58870001274" containerName="revoke-keys" Jul 19 08:13:06.667010 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:06.667002 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="75400e5a-56e3-4e6a-a8f4-2042ba9d90d6" containerName="revoke-keys" Jul 19 08:13:06.667110 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:06.667012 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="75400e5a-56e3-4e6a-a8f4-2042ba9d90d6" containerName="revoke-keys" Jul 19 08:13:06.670223 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:06.670208 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-2hv7v" Jul 19 08:13:06.672683 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:06.672663 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qbdd4\"" Jul 19 08:13:06.675860 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:06.675835 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-2hv7v"] Jul 19 08:13:06.728348 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:06.728311 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8c2a786-6163-4ec5-b523-7112e97c80e6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef124b28-2hv7v\" (UID: \"b8c2a786-6163-4ec5-b523-7112e97c80e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-2hv7v" Jul 19 08:13:06.829338 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:06.829293 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8c2a786-6163-4ec5-b523-7112e97c80e6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef124b28-2hv7v\" (UID: \"b8c2a786-6163-4ec5-b523-7112e97c80e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-2hv7v" Jul 19 08:13:06.829980 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:06.829957 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8c2a786-6163-4ec5-b523-7112e97c80e6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef124b28-2hv7v\" (UID: \"b8c2a786-6163-4ec5-b523-7112e97c80e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-2hv7v" Jul 19 08:13:06.981875 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:06.981840 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-2hv7v" Jul 19 08:13:07.119594 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:07.119539 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-2hv7v"] Jul 19 08:13:07.121821 ip-10-0-136-70 kubenswrapper[2574]: W0719 08:13:07.121792 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8c2a786_6163_4ec5_b523_7112e97c80e6.slice/crio-d5e4e10f7f7ddc910bb19998695b3248dc1431d4784881864e045eb573fa1e7e WatchSource:0}: Error finding container d5e4e10f7f7ddc910bb19998695b3248dc1431d4784881864e045eb573fa1e7e: Status 404 returned error can't find the container with id d5e4e10f7f7ddc910bb19998695b3248dc1431d4784881864e045eb573fa1e7e Jul 19 08:13:07.968412 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:07.968375 2574 generic.go:356] "Generic (PLEG): container finished" podID="b8c2a786-6163-4ec5-b523-7112e97c80e6" containerID="d44e2ab9e9a4fef20e8753cc5392f99eface8a371827793c9c4ccb87c1469de8" exitCode=7 Jul 19 08:13:07.968838 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:07.968456 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-2hv7v" event={"ID":"b8c2a786-6163-4ec5-b523-7112e97c80e6","Type":"ContainerDied","Data":"d44e2ab9e9a4fef20e8753cc5392f99eface8a371827793c9c4ccb87c1469de8"} Jul 19 08:13:07.968838 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:07.968513 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-2hv7v" event={"ID":"b8c2a786-6163-4ec5-b523-7112e97c80e6","Type":"ContainerStarted","Data":"d5e4e10f7f7ddc910bb19998695b3248dc1431d4784881864e045eb573fa1e7e"} Jul 19 08:13:07.968838 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:07.968680 2574 scope.go:117] "RemoveContainer" containerID="d44e2ab9e9a4fef20e8753cc5392f99eface8a371827793c9c4ccb87c1469de8" Jul 19 08:13:08.975274 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:08.975238 2574 generic.go:356] "Generic (PLEG): container finished" podID="b8c2a786-6163-4ec5-b523-7112e97c80e6" containerID="a41bf8055b1310c323a1767b0aba5fa13407175ddc157390d2a1273067300618" exitCode=7 Jul 19 08:13:08.975710 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:08.975330 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-2hv7v" event={"ID":"b8c2a786-6163-4ec5-b523-7112e97c80e6","Type":"ContainerDied","Data":"a41bf8055b1310c323a1767b0aba5fa13407175ddc157390d2a1273067300618"} Jul 19 08:13:08.975710 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:08.975411 2574 scope.go:117] "RemoveContainer" containerID="d44e2ab9e9a4fef20e8753cc5392f99eface8a371827793c9c4ccb87c1469de8" Jul 19 08:13:08.975710 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:08.975491 2574 scope.go:117] "RemoveContainer" containerID="a41bf8055b1310c323a1767b0aba5fa13407175ddc157390d2a1273067300618" Jul 19 08:13:08.975838 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:13:08.975767 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ef124b28-2hv7v_odh-ai-gateway-infra(b8c2a786-6163-4ec5-b523-7112e97c80e6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-2hv7v" podUID="b8c2a786-6163-4ec5-b523-7112e97c80e6" Jul 19 08:13:09.981939 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:09.981911 2574 scope.go:117] "RemoveContainer" containerID="a41bf8055b1310c323a1767b0aba5fa13407175ddc157390d2a1273067300618" Jul 19 08:13:09.982431 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:13:09.982184 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ef124b28-2hv7v_odh-ai-gateway-infra(b8c2a786-6163-4ec5-b523-7112e97c80e6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-2hv7v" podUID="b8c2a786-6163-4ec5-b523-7112e97c80e6" Jul 19 08:13:20.552857 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:20.552817 2574 scope.go:117] "RemoveContainer" containerID="8d5c0e9e71b976ca7d6025339747a9f6d76e141dc863df70935949548a6a9065" Jul 19 08:13:20.555389 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:13:20.553048 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-ef124b28-969b9ccd7-g5sk7_odh-ai-gateway-infra(0cf73e24-459f-470a-a9fd-c9b2af7a11b1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef124b28-969b9ccd7-g5sk7" podUID="0cf73e24-459f-470a-a9fd-c9b2af7a11b1" Jul 19 08:13:22.553225 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:22.553192 2574 scope.go:117] "RemoveContainer" containerID="a41bf8055b1310c323a1767b0aba5fa13407175ddc157390d2a1273067300618" Jul 19 08:13:23.048454 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:23.048418 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-2hv7v" event={"ID":"b8c2a786-6163-4ec5-b523-7112e97c80e6","Type":"ContainerStarted","Data":"5e645b03668f3c8476266ec5369bc5806dc7dc3901913df0f49579ff84d62dd4"} Jul 19 08:13:23.063348 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:23.063295 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-2hv7v" podStartSLOduration=17.063277928 podStartE2EDuration="17.063277928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 08:13:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:13:23.062177786 +0000 UTC m=+1869.416487399" watchObservedRunningTime="2026-07-19 08:13:23.063277928 +0000 UTC m=+1869.417587529" Jul 19 08:13:23.782540 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:23.782492 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-2hv7v"] Jul 19 08:13:24.055794 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:24.055699 2574 generic.go:356] "Generic (PLEG): container finished" podID="b8c2a786-6163-4ec5-b523-7112e97c80e6" containerID="5e645b03668f3c8476266ec5369bc5806dc7dc3901913df0f49579ff84d62dd4" exitCode=7 Jul 19 08:13:24.055794 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:24.055764 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-2hv7v" event={"ID":"b8c2a786-6163-4ec5-b523-7112e97c80e6","Type":"ContainerDied","Data":"5e645b03668f3c8476266ec5369bc5806dc7dc3901913df0f49579ff84d62dd4"} Jul 19 08:13:24.056040 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:24.055822 2574 scope.go:117] "RemoveContainer" containerID="a41bf8055b1310c323a1767b0aba5fa13407175ddc157390d2a1273067300618" Jul 19 08:13:24.183469 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:24.183442 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-2hv7v" Jul 19 08:13:24.288354 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:24.288319 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8c2a786-6163-4ec5-b523-7112e97c80e6-maas-api-service-ca\") pod \"b8c2a786-6163-4ec5-b523-7112e97c80e6\" (UID: \"b8c2a786-6163-4ec5-b523-7112e97c80e6\") " Jul 19 08:13:24.288743 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:24.288716 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8c2a786-6163-4ec5-b523-7112e97c80e6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b8c2a786-6163-4ec5-b523-7112e97c80e6" (UID: "b8c2a786-6163-4ec5-b523-7112e97c80e6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 08:13:24.389365 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:24.389279 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8c2a786-6163-4ec5-b523-7112e97c80e6-maas-api-service-ca\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:13:25.060999 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:25.060966 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-2hv7v" Jul 19 08:13:25.061441 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:25.061008 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-2hv7v" event={"ID":"b8c2a786-6163-4ec5-b523-7112e97c80e6","Type":"ContainerDied","Data":"d5e4e10f7f7ddc910bb19998695b3248dc1431d4784881864e045eb573fa1e7e"} Jul 19 08:13:25.061441 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:25.061067 2574 scope.go:117] "RemoveContainer" containerID="5e645b03668f3c8476266ec5369bc5806dc7dc3901913df0f49579ff84d62dd4" Jul 19 08:13:25.078319 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:25.078292 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-2hv7v"] Jul 19 08:13:25.081825 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:25.081800 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-2hv7v"] Jul 19 08:13:26.559415 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:26.559370 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8c2a786-6163-4ec5-b523-7112e97c80e6" path="/var/lib/kubelet/pods/b8c2a786-6163-4ec5-b523-7112e97c80e6/volumes" Jul 19 08:13:26.701035 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:26.700997 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:13:27.873961 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:27.873924 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:13:29.675698 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:29.675657 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 08:13:32.490408 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:32.490371 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ef124b28-969b9ccd7-g5sk7"] Jul 19 08:13:32.591082 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:32.591049 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:13:32.650561 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:32.650536 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef124b28-969b9ccd7-g5sk7" Jul 19 08:13:32.720065 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:32.720018 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:13:32.767199 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:32.767108 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0cf73e24-459f-470a-a9fd-c9b2af7a11b1-maas-api-tls\") pod \"0cf73e24-459f-470a-a9fd-c9b2af7a11b1\" (UID: \"0cf73e24-459f-470a-a9fd-c9b2af7a11b1\") " Jul 19 08:13:32.767199 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:32.767152 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cf73e24-459f-470a-a9fd-c9b2af7a11b1-openshift-service-ca\") pod \"0cf73e24-459f-470a-a9fd-c9b2af7a11b1\" (UID: \"0cf73e24-459f-470a-a9fd-c9b2af7a11b1\") " Jul 19 08:13:32.767490 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:32.767217 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9tsc6\" (UniqueName: \"kubernetes.io/projected/0cf73e24-459f-470a-a9fd-c9b2af7a11b1-kube-api-access-9tsc6\") pod \"0cf73e24-459f-470a-a9fd-c9b2af7a11b1\" (UID: \"0cf73e24-459f-470a-a9fd-c9b2af7a11b1\") " Jul 19 08:13:32.767626 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:32.767603 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cf73e24-459f-470a-a9fd-c9b2af7a11b1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0cf73e24-459f-470a-a9fd-c9b2af7a11b1" (UID: "0cf73e24-459f-470a-a9fd-c9b2af7a11b1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 08:13:32.769434 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:32.769410 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cf73e24-459f-470a-a9fd-c9b2af7a11b1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0cf73e24-459f-470a-a9fd-c9b2af7a11b1" (UID: "0cf73e24-459f-470a-a9fd-c9b2af7a11b1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 08:13:32.769528 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:32.769458 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cf73e24-459f-470a-a9fd-c9b2af7a11b1-kube-api-access-9tsc6" (OuterVolumeSpecName: "kube-api-access-9tsc6") pod "0cf73e24-459f-470a-a9fd-c9b2af7a11b1" (UID: "0cf73e24-459f-470a-a9fd-c9b2af7a11b1"). InnerVolumeSpecName "kube-api-access-9tsc6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 08:13:32.868849 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:32.868811 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tsc6\" (UniqueName: \"kubernetes.io/projected/0cf73e24-459f-470a-a9fd-c9b2af7a11b1-kube-api-access-9tsc6\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:13:32.868849 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:32.868843 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0cf73e24-459f-470a-a9fd-c9b2af7a11b1-maas-api-tls\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:13:32.868849 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:32.868852 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cf73e24-459f-470a-a9fd-c9b2af7a11b1-openshift-service-ca\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:13:33.102304 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:33.102221 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef124b28-969b9ccd7-g5sk7" Jul 19 08:13:33.102304 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:33.102266 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef124b28-969b9ccd7-g5sk7" event={"ID":"0cf73e24-459f-470a-a9fd-c9b2af7a11b1","Type":"ContainerDied","Data":"a75f0a786e801abfd003c6aebb51e419ee9425aa36656fb3a4d493a8c40ef557"} Jul 19 08:13:33.102546 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:33.102326 2574 scope.go:117] "RemoveContainer" containerID="8d5c0e9e71b976ca7d6025339747a9f6d76e141dc863df70935949548a6a9065" Jul 19 08:13:33.123377 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:33.123351 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ef124b28-969b9ccd7-g5sk7"] Jul 19 08:13:33.128500 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:33.128477 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ef124b28-969b9ccd7-g5sk7"] Jul 19 08:13:34.558780 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:34.558746 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0cf73e24-459f-470a-a9fd-c9b2af7a11b1" path="/var/lib/kubelet/pods/0cf73e24-459f-470a-a9fd-c9b2af7a11b1/volumes" Jul 19 08:13:38.114902 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:38.114812 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:13:38.172755 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:38.172719 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:13:38.370980 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:38.370893 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:13:38.495466 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:38.495428 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:13:38.611038 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:38.610998 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-1c28d394-gw-openshift-default-6d59c559cf-w9pht"] Jul 19 08:13:38.611604 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:38.611580 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8c2a786-6163-4ec5-b523-7112e97c80e6" containerName="revoke-keys" Jul 19 08:13:38.611604 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:38.611603 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8c2a786-6163-4ec5-b523-7112e97c80e6" containerName="revoke-keys" Jul 19 08:13:38.611798 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:38.611618 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8c2a786-6163-4ec5-b523-7112e97c80e6" containerName="revoke-keys" Jul 19 08:13:38.611798 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:38.611654 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8c2a786-6163-4ec5-b523-7112e97c80e6" containerName="revoke-keys" Jul 19 08:13:38.611798 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:38.611675 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cf73e24-459f-470a-a9fd-c9b2af7a11b1" containerName="maas-api" Jul 19 08:13:38.611798 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:38.611684 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cf73e24-459f-470a-a9fd-c9b2af7a11b1" containerName="maas-api" Jul 19 08:13:38.611798 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:38.611700 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cf73e24-459f-470a-a9fd-c9b2af7a11b1" containerName="maas-api" Jul 19 08:13:38.611798 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:38.611709 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cf73e24-459f-470a-a9fd-c9b2af7a11b1" containerName="maas-api" Jul 19 08:13:38.611798 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:38.611720 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cf73e24-459f-470a-a9fd-c9b2af7a11b1" containerName="maas-api" Jul 19 08:13:38.611798 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:38.611729 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cf73e24-459f-470a-a9fd-c9b2af7a11b1" containerName="maas-api" Jul 19 08:13:38.611798 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:38.611740 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75400e5a-56e3-4e6a-a8f4-2042ba9d90d6" containerName="revoke-keys" Jul 19 08:13:38.611798 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:38.611749 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="75400e5a-56e3-4e6a-a8f4-2042ba9d90d6" containerName="revoke-keys" Jul 19 08:13:38.611798 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:38.611773 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cf73e24-459f-470a-a9fd-c9b2af7a11b1" containerName="maas-api" Jul 19 08:13:38.611798 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:38.611782 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cf73e24-459f-470a-a9fd-c9b2af7a11b1" containerName="maas-api" Jul 19 08:13:38.612473 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:38.611905 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="0cf73e24-459f-470a-a9fd-c9b2af7a11b1" containerName="maas-api" Jul 19 08:13:38.612473 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:38.611918 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="75400e5a-56e3-4e6a-a8f4-2042ba9d90d6" containerName="revoke-keys" Jul 19 08:13:38.612473 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:38.611930 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8c2a786-6163-4ec5-b523-7112e97c80e6" containerName="revoke-keys" Jul 19 08:13:38.612473 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:38.611943 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="0cf73e24-459f-470a-a9fd-c9b2af7a11b1" containerName="maas-api" Jul 19 08:13:38.612473 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:38.611951 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8c2a786-6163-4ec5-b523-7112e97c80e6" containerName="revoke-keys" Jul 19 08:13:38.612473 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:38.611960 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8c2a786-6163-4ec5-b523-7112e97c80e6" containerName="revoke-keys" Jul 19 08:13:38.612473 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:38.611969 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="0cf73e24-459f-470a-a9fd-c9b2af7a11b1" containerName="maas-api" Jul 19 08:13:38.612473 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:38.611977 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="0cf73e24-459f-470a-a9fd-c9b2af7a11b1" containerName="maas-api" Jul 19 08:13:38.616864 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:38.616836 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-1c28d394-gw-openshift-default-6d59c559cf-w9pht" Jul 19 08:13:38.619647 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:38.619620 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 19 08:13:38.619974 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:38.619641 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-1c28d394-gw-openshift-default-dockercfg-lbvqt\"" Jul 19 08:13:38.629753 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:38.629721 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-1c28d394-gw-openshift-default-6d59c559cf-w9pht"] Jul 19 08:13:38.646769 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:38.646740 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:13:38.717888 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:38.717851 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21210c5f-f1fd-4853-85dc-258f9b910be5-workload-certs\") pod \"e2e-migrate-1c28d394-gw-openshift-default-6d59c559cf-w9pht\" (UID: \"21210c5f-f1fd-4853-85dc-258f9b910be5\") " pod="openshift-ingress/e2e-migrate-1c28d394-gw-openshift-default-6d59c559cf-w9pht" Jul 19 08:13:38.718073 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:38.717897 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21210c5f-f1fd-4853-85dc-258f9b910be5-credential-socket\") pod \"e2e-migrate-1c28d394-gw-openshift-default-6d59c559cf-w9pht\" (UID: \"21210c5f-f1fd-4853-85dc-258f9b910be5\") " pod="openshift-ingress/e2e-migrate-1c28d394-gw-openshift-default-6d59c559cf-w9pht" Jul 19 08:13:38.718073 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:38.717918 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21210c5f-f1fd-4853-85dc-258f9b910be5-istio-envoy\") pod \"e2e-migrate-1c28d394-gw-openshift-default-6d59c559cf-w9pht\" (UID: \"21210c5f-f1fd-4853-85dc-258f9b910be5\") " pod="openshift-ingress/e2e-migrate-1c28d394-gw-openshift-default-6d59c559cf-w9pht" Jul 19 08:13:38.718073 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:38.717960 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21210c5f-f1fd-4853-85dc-258f9b910be5-istiod-ca-cert\") pod \"e2e-migrate-1c28d394-gw-openshift-default-6d59c559cf-w9pht\" (UID: \"21210c5f-f1fd-4853-85dc-258f9b910be5\") " pod="openshift-ingress/e2e-migrate-1c28d394-gw-openshift-default-6d59c559cf-w9pht" Jul 19 08:13:38.718190 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:38.718081 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21210c5f-f1fd-4853-85dc-258f9b910be5-istio-podinfo\") pod \"e2e-migrate-1c28d394-gw-openshift-default-6d59c559cf-w9pht\" (UID: \"21210c5f-f1fd-4853-85dc-258f9b910be5\") " pod="openshift-ingress/e2e-migrate-1c28d394-gw-openshift-default-6d59c559cf-w9pht" Jul 19 08:13:38.718190 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:38.718130 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21210c5f-f1fd-4853-85dc-258f9b910be5-istio-data\") pod \"e2e-migrate-1c28d394-gw-openshift-default-6d59c559cf-w9pht\" (UID: \"21210c5f-f1fd-4853-85dc-258f9b910be5\") " pod="openshift-ingress/e2e-migrate-1c28d394-gw-openshift-default-6d59c559cf-w9pht" Jul 19 08:13:38.718190 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:38.718173 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21210c5f-f1fd-4853-85dc-258f9b910be5-istio-token\") pod \"e2e-migrate-1c28d394-gw-openshift-default-6d59c559cf-w9pht\" (UID: \"21210c5f-f1fd-4853-85dc-258f9b910be5\") " pod="openshift-ingress/e2e-migrate-1c28d394-gw-openshift-default-6d59c559cf-w9pht" Jul 19 08:13:38.718286 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:38.718230 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21210c5f-f1fd-4853-85dc-258f9b910be5-workload-socket\") pod \"e2e-migrate-1c28d394-gw-openshift-default-6d59c559cf-w9pht\" (UID: \"21210c5f-f1fd-4853-85dc-258f9b910be5\") " pod="openshift-ingress/e2e-migrate-1c28d394-gw-openshift-default-6d59c559cf-w9pht" Jul 19 08:13:38.718286 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:38.718260 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcj27\" (UniqueName: \"kubernetes.io/projected/21210c5f-f1fd-4853-85dc-258f9b910be5-kube-api-access-jcj27\") pod \"e2e-migrate-1c28d394-gw-openshift-default-6d59c559cf-w9pht\" (UID: \"21210c5f-f1fd-4853-85dc-258f9b910be5\") " pod="openshift-ingress/e2e-migrate-1c28d394-gw-openshift-default-6d59c559cf-w9pht" Jul 19 08:13:38.819681 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:38.819638 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21210c5f-f1fd-4853-85dc-258f9b910be5-workload-socket\") pod \"e2e-migrate-1c28d394-gw-openshift-default-6d59c559cf-w9pht\" (UID: \"21210c5f-f1fd-4853-85dc-258f9b910be5\") " pod="openshift-ingress/e2e-migrate-1c28d394-gw-openshift-default-6d59c559cf-w9pht" Jul 19 08:13:38.819917 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:38.819694 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcj27\" (UniqueName: \"kubernetes.io/projected/21210c5f-f1fd-4853-85dc-258f9b910be5-kube-api-access-jcj27\") pod \"e2e-migrate-1c28d394-gw-openshift-default-6d59c559cf-w9pht\" (UID: \"21210c5f-f1fd-4853-85dc-258f9b910be5\") " pod="openshift-ingress/e2e-migrate-1c28d394-gw-openshift-default-6d59c559cf-w9pht" Jul 19 08:13:38.819917 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:38.819716 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21210c5f-f1fd-4853-85dc-258f9b910be5-workload-certs\") pod \"e2e-migrate-1c28d394-gw-openshift-default-6d59c559cf-w9pht\" (UID: \"21210c5f-f1fd-4853-85dc-258f9b910be5\") " pod="openshift-ingress/e2e-migrate-1c28d394-gw-openshift-default-6d59c559cf-w9pht" Jul 19 08:13:38.819917 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:38.819752 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21210c5f-f1fd-4853-85dc-258f9b910be5-credential-socket\") pod \"e2e-migrate-1c28d394-gw-openshift-default-6d59c559cf-w9pht\" (UID: \"21210c5f-f1fd-4853-85dc-258f9b910be5\") " pod="openshift-ingress/e2e-migrate-1c28d394-gw-openshift-default-6d59c559cf-w9pht" Jul 19 08:13:38.819917 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:38.819780 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21210c5f-f1fd-4853-85dc-258f9b910be5-istio-envoy\") pod \"e2e-migrate-1c28d394-gw-openshift-default-6d59c559cf-w9pht\" (UID: \"21210c5f-f1fd-4853-85dc-258f9b910be5\") " pod="openshift-ingress/e2e-migrate-1c28d394-gw-openshift-default-6d59c559cf-w9pht" Jul 19 08:13:38.819917 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:38.819802 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21210c5f-f1fd-4853-85dc-258f9b910be5-istiod-ca-cert\") pod \"e2e-migrate-1c28d394-gw-openshift-default-6d59c559cf-w9pht\" (UID: \"21210c5f-f1fd-4853-85dc-258f9b910be5\") " pod="openshift-ingress/e2e-migrate-1c28d394-gw-openshift-default-6d59c559cf-w9pht" Jul 19 08:13:38.819917 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:38.819861 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21210c5f-f1fd-4853-85dc-258f9b910be5-istio-podinfo\") pod \"e2e-migrate-1c28d394-gw-openshift-default-6d59c559cf-w9pht\" (UID: \"21210c5f-f1fd-4853-85dc-258f9b910be5\") " pod="openshift-ingress/e2e-migrate-1c28d394-gw-openshift-default-6d59c559cf-w9pht" Jul 19 08:13:38.820295 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:38.819923 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21210c5f-f1fd-4853-85dc-258f9b910be5-istio-data\") pod \"e2e-migrate-1c28d394-gw-openshift-default-6d59c559cf-w9pht\" (UID: \"21210c5f-f1fd-4853-85dc-258f9b910be5\") " pod="openshift-ingress/e2e-migrate-1c28d394-gw-openshift-default-6d59c559cf-w9pht" Jul 19 08:13:38.820295 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:38.819946 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21210c5f-f1fd-4853-85dc-258f9b910be5-istio-token\") pod \"e2e-migrate-1c28d394-gw-openshift-default-6d59c559cf-w9pht\" (UID: \"21210c5f-f1fd-4853-85dc-258f9b910be5\") " pod="openshift-ingress/e2e-migrate-1c28d394-gw-openshift-default-6d59c559cf-w9pht" Jul 19 08:13:38.820295 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:38.820143 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21210c5f-f1fd-4853-85dc-258f9b910be5-workload-socket\") pod \"e2e-migrate-1c28d394-gw-openshift-default-6d59c559cf-w9pht\" (UID: \"21210c5f-f1fd-4853-85dc-258f9b910be5\") " pod="openshift-ingress/e2e-migrate-1c28d394-gw-openshift-default-6d59c559cf-w9pht" Jul 19 08:13:38.820614 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:38.820433 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21210c5f-f1fd-4853-85dc-258f9b910be5-workload-certs\") pod \"e2e-migrate-1c28d394-gw-openshift-default-6d59c559cf-w9pht\" (UID: \"21210c5f-f1fd-4853-85dc-258f9b910be5\") " pod="openshift-ingress/e2e-migrate-1c28d394-gw-openshift-default-6d59c559cf-w9pht" Jul 19 08:13:38.820614 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:38.820567 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21210c5f-f1fd-4853-85dc-258f9b910be5-credential-socket\") pod \"e2e-migrate-1c28d394-gw-openshift-default-6d59c559cf-w9pht\" (UID: \"21210c5f-f1fd-4853-85dc-258f9b910be5\") " pod="openshift-ingress/e2e-migrate-1c28d394-gw-openshift-default-6d59c559cf-w9pht" Jul 19 08:13:38.820614 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:38.820602 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21210c5f-f1fd-4853-85dc-258f9b910be5-istio-data\") pod \"e2e-migrate-1c28d394-gw-openshift-default-6d59c559cf-w9pht\" (UID: \"21210c5f-f1fd-4853-85dc-258f9b910be5\") " pod="openshift-ingress/e2e-migrate-1c28d394-gw-openshift-default-6d59c559cf-w9pht" Jul 19 08:13:38.820926 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:38.820907 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21210c5f-f1fd-4853-85dc-258f9b910be5-istiod-ca-cert\") pod \"e2e-migrate-1c28d394-gw-openshift-default-6d59c559cf-w9pht\" (UID: \"21210c5f-f1fd-4853-85dc-258f9b910be5\") " pod="openshift-ingress/e2e-migrate-1c28d394-gw-openshift-default-6d59c559cf-w9pht" Jul 19 08:13:38.822914 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:38.822892 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21210c5f-f1fd-4853-85dc-258f9b910be5-istio-envoy\") pod \"e2e-migrate-1c28d394-gw-openshift-default-6d59c559cf-w9pht\" (UID: \"21210c5f-f1fd-4853-85dc-258f9b910be5\") " pod="openshift-ingress/e2e-migrate-1c28d394-gw-openshift-default-6d59c559cf-w9pht" Jul 19 08:13:38.823298 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:38.823272 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21210c5f-f1fd-4853-85dc-258f9b910be5-istio-podinfo\") pod \"e2e-migrate-1c28d394-gw-openshift-default-6d59c559cf-w9pht\" (UID: \"21210c5f-f1fd-4853-85dc-258f9b910be5\") " pod="openshift-ingress/e2e-migrate-1c28d394-gw-openshift-default-6d59c559cf-w9pht" Jul 19 08:13:38.827869 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:38.827845 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21210c5f-f1fd-4853-85dc-258f9b910be5-istio-token\") pod \"e2e-migrate-1c28d394-gw-openshift-default-6d59c559cf-w9pht\" (UID: \"21210c5f-f1fd-4853-85dc-258f9b910be5\") " pod="openshift-ingress/e2e-migrate-1c28d394-gw-openshift-default-6d59c559cf-w9pht" Jul 19 08:13:38.828058 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:38.828037 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcj27\" (UniqueName: \"kubernetes.io/projected/21210c5f-f1fd-4853-85dc-258f9b910be5-kube-api-access-jcj27\") pod \"e2e-migrate-1c28d394-gw-openshift-default-6d59c559cf-w9pht\" (UID: \"21210c5f-f1fd-4853-85dc-258f9b910be5\") " pod="openshift-ingress/e2e-migrate-1c28d394-gw-openshift-default-6d59c559cf-w9pht" Jul 19 08:13:38.930843 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:38.930741 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-1c28d394-gw-openshift-default-6d59c559cf-w9pht" Jul 19 08:13:39.083623 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:39.083596 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-1c28d394-gw-openshift-default-6d59c559cf-w9pht"] Jul 19 08:13:39.085926 ip-10-0-136-70 kubenswrapper[2574]: W0719 08:13:39.085884 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21210c5f_f1fd_4853_85dc_258f9b910be5.slice/crio-d60ee9dfe3db05b4d0c5c82c2bf4f839e5a46932558400a674cf1efb66877ebf WatchSource:0}: Error finding container d60ee9dfe3db05b4d0c5c82c2bf4f839e5a46932558400a674cf1efb66877ebf: Status 404 returned error can't find the container with id d60ee9dfe3db05b4d0c5c82c2bf4f839e5a46932558400a674cf1efb66877ebf Jul 19 08:13:39.134081 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:39.134036 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-1c28d394-gw-openshift-default-6d59c559cf-w9pht" event={"ID":"21210c5f-f1fd-4853-85dc-258f9b910be5","Type":"ContainerStarted","Data":"d60ee9dfe3db05b4d0c5c82c2bf4f839e5a46932558400a674cf1efb66877ebf"} Jul 19 08:13:39.887082 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:39.887044 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:13:39.981284 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:39.981236 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-1c28d394-767864b5dd-lzkfv"] Jul 19 08:13:39.981881 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:39.981857 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cf73e24-459f-470a-a9fd-c9b2af7a11b1" containerName="maas-api" Jul 19 08:13:39.981881 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:39.981883 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cf73e24-459f-470a-a9fd-c9b2af7a11b1" containerName="maas-api" Jul 19 08:13:39.982065 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:39.981911 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8c2a786-6163-4ec5-b523-7112e97c80e6" containerName="revoke-keys" Jul 19 08:13:39.982065 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:39.981921 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8c2a786-6163-4ec5-b523-7112e97c80e6" containerName="revoke-keys" Jul 19 08:13:39.982065 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:39.982024 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="0cf73e24-459f-470a-a9fd-c9b2af7a11b1" containerName="maas-api" Jul 19 08:13:39.986866 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:39.986843 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1c28d394-767864b5dd-lzkfv" Jul 19 08:13:39.989630 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:39.989568 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 19 08:13:39.989630 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:39.989591 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-tdbkh\"" Jul 19 08:13:39.990467 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:39.990444 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-1c28d394-serving-cert\"" Jul 19 08:13:39.990624 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:39.990501 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 19 08:13:39.995634 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:39.995601 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-1c28d394-767864b5dd-lzkfv"] Jul 19 08:13:40.013056 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:40.013032 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:13:40.133404 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:40.133351 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sf5ql\" (UniqueName: \"kubernetes.io/projected/1566fae3-6df6-4396-b8f9-78c96483c6a1-kube-api-access-sf5ql\") pod \"maas-api-e2e-migrate-1c28d394-767864b5dd-lzkfv\" (UID: \"1566fae3-6df6-4396-b8f9-78c96483c6a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1c28d394-767864b5dd-lzkfv" Jul 19 08:13:40.133602 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:40.133456 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1566fae3-6df6-4396-b8f9-78c96483c6a1-openshift-service-ca\") pod \"maas-api-e2e-migrate-1c28d394-767864b5dd-lzkfv\" (UID: \"1566fae3-6df6-4396-b8f9-78c96483c6a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1c28d394-767864b5dd-lzkfv" Jul 19 08:13:40.133685 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:40.133619 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1566fae3-6df6-4396-b8f9-78c96483c6a1-maas-api-tls\") pod \"maas-api-e2e-migrate-1c28d394-767864b5dd-lzkfv\" (UID: \"1566fae3-6df6-4396-b8f9-78c96483c6a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1c28d394-767864b5dd-lzkfv" Jul 19 08:13:40.234986 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:40.234945 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1566fae3-6df6-4396-b8f9-78c96483c6a1-maas-api-tls\") pod \"maas-api-e2e-migrate-1c28d394-767864b5dd-lzkfv\" (UID: \"1566fae3-6df6-4396-b8f9-78c96483c6a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1c28d394-767864b5dd-lzkfv" Jul 19 08:13:40.235463 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:40.235026 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sf5ql\" (UniqueName: \"kubernetes.io/projected/1566fae3-6df6-4396-b8f9-78c96483c6a1-kube-api-access-sf5ql\") pod \"maas-api-e2e-migrate-1c28d394-767864b5dd-lzkfv\" (UID: \"1566fae3-6df6-4396-b8f9-78c96483c6a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1c28d394-767864b5dd-lzkfv" Jul 19 08:13:40.235463 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:40.235061 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1566fae3-6df6-4396-b8f9-78c96483c6a1-openshift-service-ca\") pod \"maas-api-e2e-migrate-1c28d394-767864b5dd-lzkfv\" (UID: \"1566fae3-6df6-4396-b8f9-78c96483c6a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1c28d394-767864b5dd-lzkfv" Jul 19 08:13:40.235463 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:13:40.235139 2574 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-1c28d394-serving-cert: secret "maas-api-e2e-migrate-1c28d394-serving-cert" not found Jul 19 08:13:40.235463 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:13:40.235218 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1566fae3-6df6-4396-b8f9-78c96483c6a1-maas-api-tls podName:1566fae3-6df6-4396-b8f9-78c96483c6a1 nodeName:}" failed. No retries permitted until 2026-07-19 08:13:40.735195615 +0000 UTC m=+1887.089505208 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1566fae3-6df6-4396-b8f9-78c96483c6a1-maas-api-tls") pod "maas-api-e2e-migrate-1c28d394-767864b5dd-lzkfv" (UID: "1566fae3-6df6-4396-b8f9-78c96483c6a1") : secret "maas-api-e2e-migrate-1c28d394-serving-cert" not found Jul 19 08:13:40.235924 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:40.235903 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1566fae3-6df6-4396-b8f9-78c96483c6a1-openshift-service-ca\") pod \"maas-api-e2e-migrate-1c28d394-767864b5dd-lzkfv\" (UID: \"1566fae3-6df6-4396-b8f9-78c96483c6a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1c28d394-767864b5dd-lzkfv" Jul 19 08:13:40.243764 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:40.243738 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sf5ql\" (UniqueName: \"kubernetes.io/projected/1566fae3-6df6-4396-b8f9-78c96483c6a1-kube-api-access-sf5ql\") pod \"maas-api-e2e-migrate-1c28d394-767864b5dd-lzkfv\" (UID: \"1566fae3-6df6-4396-b8f9-78c96483c6a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1c28d394-767864b5dd-lzkfv" Jul 19 08:13:40.324041 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:40.324005 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:13:40.495902 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:40.495805 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:13:40.601086 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:40.601044 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:13:40.740467 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:40.740414 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1566fae3-6df6-4396-b8f9-78c96483c6a1-maas-api-tls\") pod \"maas-api-e2e-migrate-1c28d394-767864b5dd-lzkfv\" (UID: \"1566fae3-6df6-4396-b8f9-78c96483c6a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1c28d394-767864b5dd-lzkfv" Jul 19 08:13:40.743700 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:40.743663 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1566fae3-6df6-4396-b8f9-78c96483c6a1-maas-api-tls\") pod \"maas-api-e2e-migrate-1c28d394-767864b5dd-lzkfv\" (UID: \"1566fae3-6df6-4396-b8f9-78c96483c6a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1c28d394-767864b5dd-lzkfv" Jul 19 08:13:40.902823 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:40.902732 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1c28d394-767864b5dd-lzkfv" Jul 19 08:13:41.060123 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:41.060093 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-1c28d394-767864b5dd-lzkfv"] Jul 19 08:13:41.951047 ip-10-0-136-70 kubenswrapper[2574]: W0719 08:13:41.951008 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1566fae3_6df6_4396_b8f9_78c96483c6a1.slice/crio-b0108e484e28eea398e068ce52d862d1224684a7138ca893a26542a313ebb7a0 WatchSource:0}: Error finding container b0108e484e28eea398e068ce52d862d1224684a7138ca893a26542a313ebb7a0: Status 404 returned error can't find the container with id b0108e484e28eea398e068ce52d862d1224684a7138ca893a26542a313ebb7a0 Jul 19 08:13:42.014127 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:42.014090 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 19 08:13:42.014213 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:42.014165 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 19 08:13:42.014213 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:42.014201 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 19 08:13:42.154497 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:42.154459 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1c28d394-767864b5dd-lzkfv" event={"ID":"1566fae3-6df6-4396-b8f9-78c96483c6a1","Type":"ContainerStarted","Data":"b0108e484e28eea398e068ce52d862d1224684a7138ca893a26542a313ebb7a0"} Jul 19 08:13:42.156361 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:42.156329 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-1c28d394-gw-openshift-default-6d59c559cf-w9pht" event={"ID":"21210c5f-f1fd-4853-85dc-258f9b910be5","Type":"ContainerStarted","Data":"34da24aeb523485e34da6562a4efd956abefc6fa3a4431886eb9c02a66fcaf36"} Jul 19 08:13:42.177914 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:42.177861 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-1c28d394-gw-openshift-default-6d59c559cf-w9pht" podStartSLOduration=1.252036894 podStartE2EDuration="4.177842803s" totalImagesPullingTime="2.925805909s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 08:13:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:13:42.175028822 +0000 UTC m=+1888.529338425" watchObservedRunningTime="2026-07-19 08:13:42.177842803 +0000 UTC m=+1888.532152403" Jul 19 08:13:42.931771 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:42.931717 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-1c28d394-gw-openshift-default-6d59c559cf-w9pht" Jul 19 08:13:43.170337 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:43.165850 2574 generic.go:356] "Generic (PLEG): container finished" podID="1566fae3-6df6-4396-b8f9-78c96483c6a1" containerID="8858cfac0bbc26a18cabebe1c1d073d07412083bbb66c6f98d0023f966319cfe" exitCode=1 Jul 19 08:13:43.170337 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:43.166297 2574 scope.go:117] "RemoveContainer" containerID="8858cfac0bbc26a18cabebe1c1d073d07412083bbb66c6f98d0023f966319cfe" Jul 19 08:13:43.170337 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:43.166870 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1c28d394-767864b5dd-lzkfv" event={"ID":"1566fae3-6df6-4396-b8f9-78c96483c6a1","Type":"ContainerDied","Data":"8858cfac0bbc26a18cabebe1c1d073d07412083bbb66c6f98d0023f966319cfe"} Jul 19 08:13:43.171218 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:43.171009 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:13:43.270604 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:43.270567 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:13:43.376416 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:43.376376 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 08:13:43.936642 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:43.936610 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-1c28d394-gw-openshift-default-6d59c559cf-w9pht" Jul 19 08:13:44.173480 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:44.173446 2574 generic.go:356] "Generic (PLEG): container finished" podID="1566fae3-6df6-4396-b8f9-78c96483c6a1" containerID="bc8b921ea502115b3c9aacee65e170710c09930f36b6189d0fcf6c5a3acad356" exitCode=1 Jul 19 08:13:44.173993 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:44.173539 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1c28d394-767864b5dd-lzkfv" event={"ID":"1566fae3-6df6-4396-b8f9-78c96483c6a1","Type":"ContainerDied","Data":"bc8b921ea502115b3c9aacee65e170710c09930f36b6189d0fcf6c5a3acad356"} Jul 19 08:13:44.173993 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:44.173601 2574 scope.go:117] "RemoveContainer" containerID="8858cfac0bbc26a18cabebe1c1d073d07412083bbb66c6f98d0023f966319cfe" Jul 19 08:13:44.173993 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:44.173881 2574 scope.go:117] "RemoveContainer" containerID="bc8b921ea502115b3c9aacee65e170710c09930f36b6189d0fcf6c5a3acad356" Jul 19 08:13:44.173993 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:44.173903 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-1c28d394-gw-openshift-default-6d59c559cf-w9pht" Jul 19 08:13:44.174197 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:13:44.174096 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-migrate-1c28d394-767864b5dd-lzkfv_odh-ai-gateway-infra(1566fae3-6df6-4396-b8f9-78c96483c6a1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1c28d394-767864b5dd-lzkfv" podUID="1566fae3-6df6-4396-b8f9-78c96483c6a1" Jul 19 08:13:44.175082 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:44.175072 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-1c28d394-gw-openshift-default-6d59c559cf-w9pht" Jul 19 08:13:44.388911 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:44.388816 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-ps579"] Jul 19 08:13:44.393683 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:44.393659 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-ps579" Jul 19 08:13:44.396160 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:44.396138 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qbdd4\"" Jul 19 08:13:44.400188 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:44.400163 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-ps579"] Jul 19 08:13:44.577465 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:44.577428 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d813d49-30ab-434d-a246-9734ab3cf5fb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1c28d394-ps579\" (UID: \"2d813d49-30ab-434d-a246-9734ab3cf5fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-ps579" Jul 19 08:13:44.678725 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:44.678618 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d813d49-30ab-434d-a246-9734ab3cf5fb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1c28d394-ps579\" (UID: \"2d813d49-30ab-434d-a246-9734ab3cf5fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-ps579" Jul 19 08:13:44.679397 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:44.679375 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d813d49-30ab-434d-a246-9734ab3cf5fb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1c28d394-ps579\" (UID: \"2d813d49-30ab-434d-a246-9734ab3cf5fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-ps579" Jul 19 08:13:44.705185 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:44.705141 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-ps579" Jul 19 08:13:44.895148 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:44.895120 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-ps579"] Jul 19 08:13:44.897340 ip-10-0-136-70 kubenswrapper[2574]: W0719 08:13:44.897304 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d813d49_30ab_434d_a246_9734ab3cf5fb.slice/crio-61c559aae6dca4fe1fd5b0057a6e5fcea38780c527750bae921ad3b9175a8535 WatchSource:0}: Error finding container 61c559aae6dca4fe1fd5b0057a6e5fcea38780c527750bae921ad3b9175a8535: Status 404 returned error can't find the container with id 61c559aae6dca4fe1fd5b0057a6e5fcea38780c527750bae921ad3b9175a8535 Jul 19 08:13:45.180611 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:45.180582 2574 scope.go:117] "RemoveContainer" containerID="bc8b921ea502115b3c9aacee65e170710c09930f36b6189d0fcf6c5a3acad356" Jul 19 08:13:45.181102 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:13:45.180834 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-migrate-1c28d394-767864b5dd-lzkfv_odh-ai-gateway-infra(1566fae3-6df6-4396-b8f9-78c96483c6a1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1c28d394-767864b5dd-lzkfv" podUID="1566fae3-6df6-4396-b8f9-78c96483c6a1" Jul 19 08:13:45.181741 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:45.181715 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-ps579" event={"ID":"2d813d49-30ab-434d-a246-9734ab3cf5fb","Type":"ContainerStarted","Data":"61c559aae6dca4fe1fd5b0057a6e5fcea38780c527750bae921ad3b9175a8535"} Jul 19 08:13:45.903015 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:45.902975 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1c28d394-767864b5dd-lzkfv" Jul 19 08:13:46.187730 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:46.187642 2574 generic.go:356] "Generic (PLEG): container finished" podID="2d813d49-30ab-434d-a246-9734ab3cf5fb" containerID="1b9d8d2f303229bc618d5b9eff17970f216de16472476c643807aa99ce3e328f" exitCode=7 Jul 19 08:13:46.188118 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:46.187728 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-ps579" event={"ID":"2d813d49-30ab-434d-a246-9734ab3cf5fb","Type":"ContainerDied","Data":"1b9d8d2f303229bc618d5b9eff17970f216de16472476c643807aa99ce3e328f"} Jul 19 08:13:46.188118 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:46.187924 2574 scope.go:117] "RemoveContainer" containerID="1b9d8d2f303229bc618d5b9eff17970f216de16472476c643807aa99ce3e328f" Jul 19 08:13:46.188236 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:46.188214 2574 scope.go:117] "RemoveContainer" containerID="bc8b921ea502115b3c9aacee65e170710c09930f36b6189d0fcf6c5a3acad356" Jul 19 08:13:46.188388 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:13:46.188362 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-migrate-1c28d394-767864b5dd-lzkfv_odh-ai-gateway-infra(1566fae3-6df6-4396-b8f9-78c96483c6a1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1c28d394-767864b5dd-lzkfv" podUID="1566fae3-6df6-4396-b8f9-78c96483c6a1" Jul 19 08:13:46.658011 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:46.657979 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:13:47.194362 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:47.194325 2574 generic.go:356] "Generic (PLEG): container finished" podID="2d813d49-30ab-434d-a246-9734ab3cf5fb" containerID="21347686a102a6d56f059c73657bc884ca6a338674b2ecc3f76248094d4b8ca8" exitCode=7 Jul 19 08:13:47.194925 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:47.194371 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-ps579" event={"ID":"2d813d49-30ab-434d-a246-9734ab3cf5fb","Type":"ContainerDied","Data":"21347686a102a6d56f059c73657bc884ca6a338674b2ecc3f76248094d4b8ca8"} Jul 19 08:13:47.194925 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:47.194434 2574 scope.go:117] "RemoveContainer" containerID="1b9d8d2f303229bc618d5b9eff17970f216de16472476c643807aa99ce3e328f" Jul 19 08:13:47.194925 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:47.194680 2574 scope.go:117] "RemoveContainer" containerID="21347686a102a6d56f059c73657bc884ca6a338674b2ecc3f76248094d4b8ca8" Jul 19 08:13:47.195083 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:13:47.194928 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-1c28d394-ps579_odh-ai-gateway-infra(2d813d49-30ab-434d-a246-9734ab3cf5fb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-ps579" podUID="2d813d49-30ab-434d-a246-9734ab3cf5fb" Jul 19 08:13:48.201014 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:48.200979 2574 scope.go:117] "RemoveContainer" containerID="21347686a102a6d56f059c73657bc884ca6a338674b2ecc3f76248094d4b8ca8" Jul 19 08:13:48.201417 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:13:48.201177 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-1c28d394-ps579_odh-ai-gateway-infra(2d813d49-30ab-434d-a246-9734ab3cf5fb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-ps579" podUID="2d813d49-30ab-434d-a246-9734ab3cf5fb" Jul 19 08:13:49.501672 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:49.501632 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:13:49.603304 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:49.603226 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:13:50.903884 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:50.903839 2574 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1c28d394-767864b5dd-lzkfv" Jul 19 08:13:50.904299 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:50.904282 2574 scope.go:117] "RemoveContainer" containerID="bc8b921ea502115b3c9aacee65e170710c09930f36b6189d0fcf6c5a3acad356" Jul 19 08:13:50.904473 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:13:50.904453 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-migrate-1c28d394-767864b5dd-lzkfv_odh-ai-gateway-infra(1566fae3-6df6-4396-b8f9-78c96483c6a1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1c28d394-767864b5dd-lzkfv" podUID="1566fae3-6df6-4396-b8f9-78c96483c6a1" Jul 19 08:13:52.223759 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:52.223714 2574 generic.go:356] "Generic (PLEG): container finished" podID="21210c5f-f1fd-4853-85dc-258f9b910be5" containerID="34da24aeb523485e34da6562a4efd956abefc6fa3a4431886eb9c02a66fcaf36" exitCode=0 Jul 19 08:13:52.224228 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:52.223781 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-1c28d394-gw-openshift-default-6d59c559cf-w9pht" event={"ID":"21210c5f-f1fd-4853-85dc-258f9b910be5","Type":"ContainerDied","Data":"34da24aeb523485e34da6562a4efd956abefc6fa3a4431886eb9c02a66fcaf36"} Jul 19 08:13:52.224228 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:52.224184 2574 scope.go:117] "RemoveContainer" containerID="34da24aeb523485e34da6562a4efd956abefc6fa3a4431886eb9c02a66fcaf36" Jul 19 08:13:52.225347 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:52.225310 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 19 08:13:52.225459 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:52.225406 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 19 08:13:52.225621 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:52.225482 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 19 08:13:53.236342 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:53.236293 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-1c28d394-gw-openshift-default-6d59c559cf-w9pht" event={"ID":"21210c5f-f1fd-4853-85dc-258f9b910be5","Type":"ContainerStarted","Data":"acef55cb73d6c0ebe6d0258c6d1657d721b7c1f790a3e3cce7f56c8a9d4d76f5"} Jul 19 08:13:53.932735 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:53.931872 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-1c28d394-gw-openshift-default-6d59c559cf-w9pht" Jul 19 08:13:53.932735 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:53.932318 2574 patch_prober.go:28] interesting pod/e2e-migrate-1c28d394-gw-openshift-default-6d59c559cf-w9pht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Jul 19 08:13:53.932735 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:53.932388 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-1c28d394-gw-openshift-default-6d59c559cf-w9pht" podUID="21210c5f-f1fd-4853-85dc-258f9b910be5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Jul 19 08:13:54.932443 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:54.931936 2574 patch_prober.go:28] interesting pod/e2e-migrate-1c28d394-gw-openshift-default-6d59c559cf-w9pht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Jul 19 08:13:54.932443 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:54.932023 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-1c28d394-gw-openshift-default-6d59c559cf-w9pht" podUID="21210c5f-f1fd-4853-85dc-258f9b910be5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Jul 19 08:13:55.931503 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:55.931458 2574 patch_prober.go:28] interesting pod/e2e-migrate-1c28d394-gw-openshift-default-6d59c559cf-w9pht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Jul 19 08:13:55.931851 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:55.931818 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-1c28d394-gw-openshift-default-6d59c559cf-w9pht" podUID="21210c5f-f1fd-4853-85dc-258f9b910be5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Jul 19 08:13:56.931724 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:56.931681 2574 patch_prober.go:28] interesting pod/e2e-migrate-1c28d394-gw-openshift-default-6d59c559cf-w9pht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Jul 19 08:13:56.932360 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:56.932324 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-1c28d394-gw-openshift-default-6d59c559cf-w9pht" podUID="21210c5f-f1fd-4853-85dc-258f9b910be5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Jul 19 08:13:57.935203 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:57.935169 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-1c28d394-gw-openshift-default-6d59c559cf-w9pht" Jul 19 08:13:57.935631 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:57.935508 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-1c28d394-gw-openshift-default-6d59c559cf-w9pht" Jul 19 08:13:57.936144 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:13:57.936123 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-1c28d394-gw-openshift-default-6d59c559cf-w9pht" Jul 19 08:14:01.553026 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:01.552989 2574 scope.go:117] "RemoveContainer" containerID="21347686a102a6d56f059c73657bc884ca6a338674b2ecc3f76248094d4b8ca8" Jul 19 08:14:02.288577 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:02.288510 2574 generic.go:356] "Generic (PLEG): container finished" podID="2d813d49-30ab-434d-a246-9734ab3cf5fb" containerID="ec9c893c95991f01531da14f834eb01ebed79a440444bcc87871e75604f691cb" exitCode=7 Jul 19 08:14:02.288577 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:02.288559 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-ps579" event={"ID":"2d813d49-30ab-434d-a246-9734ab3cf5fb","Type":"ContainerDied","Data":"ec9c893c95991f01531da14f834eb01ebed79a440444bcc87871e75604f691cb"} Jul 19 08:14:02.288846 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:02.288621 2574 scope.go:117] "RemoveContainer" containerID="21347686a102a6d56f059c73657bc884ca6a338674b2ecc3f76248094d4b8ca8" Jul 19 08:14:02.288927 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:02.288911 2574 scope.go:117] "RemoveContainer" containerID="ec9c893c95991f01531da14f834eb01ebed79a440444bcc87871e75604f691cb" Jul 19 08:14:02.289175 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:14:02.289153 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-1c28d394-ps579_odh-ai-gateway-infra(2d813d49-30ab-434d-a246-9734ab3cf5fb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-ps579" podUID="2d813d49-30ab-434d-a246-9734ab3cf5fb" Jul 19 08:14:02.333780 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:02.333742 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:14:02.435583 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:02.435496 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:14:03.315646 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:03.315608 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-ps579"] Jul 19 08:14:03.466471 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:03.466443 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-ps579" Jul 19 08:14:03.555276 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:03.555238 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d813d49-30ab-434d-a246-9734ab3cf5fb-maas-api-service-ca\") pod \"2d813d49-30ab-434d-a246-9734ab3cf5fb\" (UID: \"2d813d49-30ab-434d-a246-9734ab3cf5fb\") " Jul 19 08:14:03.555624 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:03.555598 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d813d49-30ab-434d-a246-9734ab3cf5fb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2d813d49-30ab-434d-a246-9734ab3cf5fb" (UID: "2d813d49-30ab-434d-a246-9734ab3cf5fb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 08:14:03.656566 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:03.656446 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d813d49-30ab-434d-a246-9734ab3cf5fb-maas-api-service-ca\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:14:04.302238 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:04.302206 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-ps579" Jul 19 08:14:04.302238 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:04.302222 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-ps579" event={"ID":"2d813d49-30ab-434d-a246-9734ab3cf5fb","Type":"ContainerDied","Data":"61c559aae6dca4fe1fd5b0057a6e5fcea38780c527750bae921ad3b9175a8535"} Jul 19 08:14:04.302549 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:04.302276 2574 scope.go:117] "RemoveContainer" containerID="ec9c893c95991f01531da14f834eb01ebed79a440444bcc87871e75604f691cb" Jul 19 08:14:04.322830 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:04.322805 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-ps579"] Jul 19 08:14:04.328739 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:04.328717 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-ps579"] Jul 19 08:14:04.563732 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:04.563644 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d813d49-30ab-434d-a246-9734ab3cf5fb" path="/var/lib/kubelet/pods/2d813d49-30ab-434d-a246-9734ab3cf5fb/volumes" Jul 19 08:14:05.552716 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:05.552682 2574 scope.go:117] "RemoveContainer" containerID="bc8b921ea502115b3c9aacee65e170710c09930f36b6189d0fcf6c5a3acad356" Jul 19 08:14:06.315463 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:06.315359 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1c28d394-767864b5dd-lzkfv" event={"ID":"1566fae3-6df6-4396-b8f9-78c96483c6a1","Type":"ContainerStarted","Data":"b0a709a3c3252a770712025ba90074fa2d87dd5fba40624b088f9b1d098e9d58"} Jul 19 08:14:06.315701 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:06.315682 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1c28d394-767864b5dd-lzkfv" Jul 19 08:14:06.332662 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:06.332609 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1c28d394-767864b5dd-lzkfv" podStartSLOduration=25.747145934 podStartE2EDuration="27.332592641s" totalImagesPullingTime="1.585446707s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 08:13:39 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:14:06.3313009 +0000 UTC m=+1912.685610503" watchObservedRunningTime="2026-07-19 08:14:06.332592641 +0000 UTC m=+1912.686902219" Jul 19 08:14:07.322014 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:07.321975 2574 generic.go:356] "Generic (PLEG): container finished" podID="1566fae3-6df6-4396-b8f9-78c96483c6a1" containerID="b0a709a3c3252a770712025ba90074fa2d87dd5fba40624b088f9b1d098e9d58" exitCode=1 Jul 19 08:14:07.322454 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:07.322060 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1c28d394-767864b5dd-lzkfv" event={"ID":"1566fae3-6df6-4396-b8f9-78c96483c6a1","Type":"ContainerDied","Data":"b0a709a3c3252a770712025ba90074fa2d87dd5fba40624b088f9b1d098e9d58"} Jul 19 08:14:07.322454 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:07.322114 2574 scope.go:117] "RemoveContainer" containerID="bc8b921ea502115b3c9aacee65e170710c09930f36b6189d0fcf6c5a3acad356" Jul 19 08:14:07.322454 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:07.322283 2574 scope.go:117] "RemoveContainer" containerID="b0a709a3c3252a770712025ba90074fa2d87dd5fba40624b088f9b1d098e9d58" Jul 19 08:14:07.322607 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:14:07.322481 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-migrate-1c28d394-767864b5dd-lzkfv_odh-ai-gateway-infra(1566fae3-6df6-4396-b8f9-78c96483c6a1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1c28d394-767864b5dd-lzkfv" podUID="1566fae3-6df6-4396-b8f9-78c96483c6a1" Jul 19 08:14:08.329086 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:08.329055 2574 scope.go:117] "RemoveContainer" containerID="b0a709a3c3252a770712025ba90074fa2d87dd5fba40624b088f9b1d098e9d58" Jul 19 08:14:08.329549 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:14:08.329239 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-migrate-1c28d394-767864b5dd-lzkfv_odh-ai-gateway-infra(1566fae3-6df6-4396-b8f9-78c96483c6a1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1c28d394-767864b5dd-lzkfv" podUID="1566fae3-6df6-4396-b8f9-78c96483c6a1" Jul 19 08:14:10.903825 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:10.903773 2574 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1c28d394-767864b5dd-lzkfv" Jul 19 08:14:10.904258 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:10.904209 2574 scope.go:117] "RemoveContainer" containerID="b0a709a3c3252a770712025ba90074fa2d87dd5fba40624b088f9b1d098e9d58" Jul 19 08:14:10.904397 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:14:10.904376 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-migrate-1c28d394-767864b5dd-lzkfv_odh-ai-gateway-infra(1566fae3-6df6-4396-b8f9-78c96483c6a1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1c28d394-767864b5dd-lzkfv" podUID="1566fae3-6df6-4396-b8f9-78c96483c6a1" Jul 19 08:14:23.553353 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:23.553318 2574 scope.go:117] "RemoveContainer" containerID="b0a709a3c3252a770712025ba90074fa2d87dd5fba40624b088f9b1d098e9d58" Jul 19 08:14:23.553834 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:14:23.553567 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-migrate-1c28d394-767864b5dd-lzkfv_odh-ai-gateway-infra(1566fae3-6df6-4396-b8f9-78c96483c6a1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1c28d394-767864b5dd-lzkfv" podUID="1566fae3-6df6-4396-b8f9-78c96483c6a1" Jul 19 08:14:36.552836 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:36.552807 2574 scope.go:117] "RemoveContainer" containerID="b0a709a3c3252a770712025ba90074fa2d87dd5fba40624b088f9b1d098e9d58" Jul 19 08:14:37.484634 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:37.484597 2574 generic.go:356] "Generic (PLEG): container finished" podID="1566fae3-6df6-4396-b8f9-78c96483c6a1" containerID="74e228b1af3726a3c43bb9d3533b2a942959cdf03f04fd0a8e6afd85cc650cfc" exitCode=1 Jul 19 08:14:37.484805 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:37.484671 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1c28d394-767864b5dd-lzkfv" event={"ID":"1566fae3-6df6-4396-b8f9-78c96483c6a1","Type":"ContainerDied","Data":"74e228b1af3726a3c43bb9d3533b2a942959cdf03f04fd0a8e6afd85cc650cfc"} Jul 19 08:14:37.484805 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:37.484735 2574 scope.go:117] "RemoveContainer" containerID="b0a709a3c3252a770712025ba90074fa2d87dd5fba40624b088f9b1d098e9d58" Jul 19 08:14:37.485092 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:37.485070 2574 scope.go:117] "RemoveContainer" containerID="74e228b1af3726a3c43bb9d3533b2a942959cdf03f04fd0a8e6afd85cc650cfc" Jul 19 08:14:37.485340 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:14:37.485319 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-migrate-1c28d394-767864b5dd-lzkfv_odh-ai-gateway-infra(1566fae3-6df6-4396-b8f9-78c96483c6a1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1c28d394-767864b5dd-lzkfv" podUID="1566fae3-6df6-4396-b8f9-78c96483c6a1" Jul 19 08:14:40.903142 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:40.903106 2574 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1c28d394-767864b5dd-lzkfv" Jul 19 08:14:40.903623 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:40.903484 2574 scope.go:117] "RemoveContainer" containerID="74e228b1af3726a3c43bb9d3533b2a942959cdf03f04fd0a8e6afd85cc650cfc" Jul 19 08:14:40.903719 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:14:40.903698 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-migrate-1c28d394-767864b5dd-lzkfv_odh-ai-gateway-infra(1566fae3-6df6-4396-b8f9-78c96483c6a1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1c28d394-767864b5dd-lzkfv" podUID="1566fae3-6df6-4396-b8f9-78c96483c6a1" Jul 19 08:14:41.316454 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:41.316421 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1c28d394-767864b5dd-lzkfv" Jul 19 08:14:41.506129 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:41.506100 2574 scope.go:117] "RemoveContainer" containerID="74e228b1af3726a3c43bb9d3533b2a942959cdf03f04fd0a8e6afd85cc650cfc" Jul 19 08:14:41.506313 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:14:41.506293 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-migrate-1c28d394-767864b5dd-lzkfv_odh-ai-gateway-infra(1566fae3-6df6-4396-b8f9-78c96483c6a1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1c28d394-767864b5dd-lzkfv" podUID="1566fae3-6df6-4396-b8f9-78c96483c6a1" Jul 19 08:14:44.515474 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:44.515442 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-1c28d394-gw-openshift-default-6d59c559cf-w9pht"] Jul 19 08:14:44.515950 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:44.515675 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-1c28d394-gw-openshift-default-6d59c559cf-w9pht" podUID="21210c5f-f1fd-4853-85dc-258f9b910be5" containerName="istio-proxy" containerID="cri-o://acef55cb73d6c0ebe6d0258c6d1657d721b7c1f790a3e3cce7f56c8a9d4d76f5" gracePeriod=30 Jul 19 08:14:44.541720 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:44.541688 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:14:44.645411 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:44.645372 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:14:47.466357 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:47.466312 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 08:14:49.658380 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:49.658343 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-1c28d394-767864b5dd-lzkfv"] Jul 19 08:14:49.727944 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:49.727900 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:14:49.817309 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:49.817280 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-1c28d394-gw-openshift-default-6d59c559cf-w9pht" Jul 19 08:14:49.819927 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:49.819889 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1c28d394-767864b5dd-lzkfv" Jul 19 08:14:49.861270 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:49.861238 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21210c5f-f1fd-4853-85dc-258f9b910be5-istio-token\") pod \"21210c5f-f1fd-4853-85dc-258f9b910be5\" (UID: \"21210c5f-f1fd-4853-85dc-258f9b910be5\") " Jul 19 08:14:49.861270 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:49.861285 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21210c5f-f1fd-4853-85dc-258f9b910be5-istio-envoy\") pod \"21210c5f-f1fd-4853-85dc-258f9b910be5\" (UID: \"21210c5f-f1fd-4853-85dc-258f9b910be5\") " Jul 19 08:14:49.861546 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:49.861313 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21210c5f-f1fd-4853-85dc-258f9b910be5-workload-certs\") pod \"21210c5f-f1fd-4853-85dc-258f9b910be5\" (UID: \"21210c5f-f1fd-4853-85dc-258f9b910be5\") " Jul 19 08:14:49.861546 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:49.861335 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21210c5f-f1fd-4853-85dc-258f9b910be5-istiod-ca-cert\") pod \"21210c5f-f1fd-4853-85dc-258f9b910be5\" (UID: \"21210c5f-f1fd-4853-85dc-258f9b910be5\") " Jul 19 08:14:49.861546 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:49.861355 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1566fae3-6df6-4396-b8f9-78c96483c6a1-openshift-service-ca\") pod \"1566fae3-6df6-4396-b8f9-78c96483c6a1\" (UID: \"1566fae3-6df6-4396-b8f9-78c96483c6a1\") " Jul 19 08:14:49.861717 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:49.861659 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21210c5f-f1fd-4853-85dc-258f9b910be5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "21210c5f-f1fd-4853-85dc-258f9b910be5" (UID: "21210c5f-f1fd-4853-85dc-258f9b910be5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 08:14:49.861717 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:49.861676 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21210c5f-f1fd-4853-85dc-258f9b910be5-istio-podinfo\") pod \"21210c5f-f1fd-4853-85dc-258f9b910be5\" (UID: \"21210c5f-f1fd-4853-85dc-258f9b910be5\") " Jul 19 08:14:49.861832 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:49.861713 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1566fae3-6df6-4396-b8f9-78c96483c6a1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1566fae3-6df6-4396-b8f9-78c96483c6a1" (UID: "1566fae3-6df6-4396-b8f9-78c96483c6a1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 08:14:49.861832 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:49.861767 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21210c5f-f1fd-4853-85dc-258f9b910be5-istio-data\") pod \"21210c5f-f1fd-4853-85dc-258f9b910be5\" (UID: \"21210c5f-f1fd-4853-85dc-258f9b910be5\") " Jul 19 08:14:49.861832 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:49.861794 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jcj27\" (UniqueName: \"kubernetes.io/projected/21210c5f-f1fd-4853-85dc-258f9b910be5-kube-api-access-jcj27\") pod \"21210c5f-f1fd-4853-85dc-258f9b910be5\" (UID: \"21210c5f-f1fd-4853-85dc-258f9b910be5\") " Jul 19 08:14:49.861832 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:49.861828 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1566fae3-6df6-4396-b8f9-78c96483c6a1-maas-api-tls\") pod \"1566fae3-6df6-4396-b8f9-78c96483c6a1\" (UID: \"1566fae3-6df6-4396-b8f9-78c96483c6a1\") " Jul 19 08:14:49.862050 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:49.861892 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21210c5f-f1fd-4853-85dc-258f9b910be5-workload-socket\") pod \"21210c5f-f1fd-4853-85dc-258f9b910be5\" (UID: \"21210c5f-f1fd-4853-85dc-258f9b910be5\") " Jul 19 08:14:49.862050 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:49.861930 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21210c5f-f1fd-4853-85dc-258f9b910be5-credential-socket\") pod \"21210c5f-f1fd-4853-85dc-258f9b910be5\" (UID: \"21210c5f-f1fd-4853-85dc-258f9b910be5\") " Jul 19 08:14:49.862050 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:49.861963 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sf5ql\" (UniqueName: \"kubernetes.io/projected/1566fae3-6df6-4396-b8f9-78c96483c6a1-kube-api-access-sf5ql\") pod \"1566fae3-6df6-4396-b8f9-78c96483c6a1\" (UID: \"1566fae3-6df6-4396-b8f9-78c96483c6a1\") " Jul 19 08:14:49.862768 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:49.862249 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21210c5f-f1fd-4853-85dc-258f9b910be5-workload-certs\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:14:49.862768 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:49.862270 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21210c5f-f1fd-4853-85dc-258f9b910be5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "21210c5f-f1fd-4853-85dc-258f9b910be5" (UID: "21210c5f-f1fd-4853-85dc-258f9b910be5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 08:14:49.862768 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:49.862274 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1566fae3-6df6-4396-b8f9-78c96483c6a1-openshift-service-ca\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:14:49.862768 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:49.862459 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21210c5f-f1fd-4853-85dc-258f9b910be5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "21210c5f-f1fd-4853-85dc-258f9b910be5" (UID: "21210c5f-f1fd-4853-85dc-258f9b910be5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 08:14:49.862768 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:49.862693 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21210c5f-f1fd-4853-85dc-258f9b910be5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "21210c5f-f1fd-4853-85dc-258f9b910be5" (UID: "21210c5f-f1fd-4853-85dc-258f9b910be5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 08:14:49.863158 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:49.862799 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21210c5f-f1fd-4853-85dc-258f9b910be5-istio-data" (OuterVolumeSpecName: "istio-data") pod "21210c5f-f1fd-4853-85dc-258f9b910be5" (UID: "21210c5f-f1fd-4853-85dc-258f9b910be5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 08:14:49.864064 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:49.864016 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21210c5f-f1fd-4853-85dc-258f9b910be5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "21210c5f-f1fd-4853-85dc-258f9b910be5" (UID: "21210c5f-f1fd-4853-85dc-258f9b910be5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 08:14:49.864176 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:49.864150 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21210c5f-f1fd-4853-85dc-258f9b910be5-istio-token" (OuterVolumeSpecName: "istio-token") pod "21210c5f-f1fd-4853-85dc-258f9b910be5" (UID: "21210c5f-f1fd-4853-85dc-258f9b910be5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 08:14:49.865281 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:49.865024 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21210c5f-f1fd-4853-85dc-258f9b910be5-kube-api-access-jcj27" (OuterVolumeSpecName: "kube-api-access-jcj27") pod "21210c5f-f1fd-4853-85dc-258f9b910be5" (UID: "21210c5f-f1fd-4853-85dc-258f9b910be5"). InnerVolumeSpecName "kube-api-access-jcj27". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 08:14:49.865342 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:49.865306 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1566fae3-6df6-4396-b8f9-78c96483c6a1-kube-api-access-sf5ql" (OuterVolumeSpecName: "kube-api-access-sf5ql") pod "1566fae3-6df6-4396-b8f9-78c96483c6a1" (UID: "1566fae3-6df6-4396-b8f9-78c96483c6a1"). InnerVolumeSpecName "kube-api-access-sf5ql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 08:14:49.866568 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:49.866500 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1566fae3-6df6-4396-b8f9-78c96483c6a1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1566fae3-6df6-4396-b8f9-78c96483c6a1" (UID: "1566fae3-6df6-4396-b8f9-78c96483c6a1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 08:14:49.866678 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:49.866654 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/21210c5f-f1fd-4853-85dc-258f9b910be5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "21210c5f-f1fd-4853-85dc-258f9b910be5" (UID: "21210c5f-f1fd-4853-85dc-258f9b910be5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 19 08:14:49.963194 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:49.963149 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21210c5f-f1fd-4853-85dc-258f9b910be5-workload-socket\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:14:49.963194 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:49.963187 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21210c5f-f1fd-4853-85dc-258f9b910be5-credential-socket\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:14:49.963194 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:49.963202 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sf5ql\" (UniqueName: \"kubernetes.io/projected/1566fae3-6df6-4396-b8f9-78c96483c6a1-kube-api-access-sf5ql\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:14:49.963467 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:49.963217 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21210c5f-f1fd-4853-85dc-258f9b910be5-istio-token\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:14:49.963467 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:49.963228 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21210c5f-f1fd-4853-85dc-258f9b910be5-istio-envoy\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:14:49.963467 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:49.963238 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21210c5f-f1fd-4853-85dc-258f9b910be5-istiod-ca-cert\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:14:49.963467 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:49.963248 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21210c5f-f1fd-4853-85dc-258f9b910be5-istio-podinfo\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:14:49.963467 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:49.963258 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21210c5f-f1fd-4853-85dc-258f9b910be5-istio-data\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:14:49.963467 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:49.963268 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcj27\" (UniqueName: \"kubernetes.io/projected/21210c5f-f1fd-4853-85dc-258f9b910be5-kube-api-access-jcj27\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:14:49.963467 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:49.963279 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1566fae3-6df6-4396-b8f9-78c96483c6a1-maas-api-tls\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:14:50.541486 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:50.541445 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:14:50.550579 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:50.550554 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1c28d394-767864b5dd-lzkfv" Jul 19 08:14:50.550579 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:50.550570 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1c28d394-767864b5dd-lzkfv" event={"ID":"1566fae3-6df6-4396-b8f9-78c96483c6a1","Type":"ContainerDied","Data":"b0108e484e28eea398e068ce52d862d1224684a7138ca893a26542a313ebb7a0"} Jul 19 08:14:50.550793 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:50.550636 2574 scope.go:117] "RemoveContainer" containerID="74e228b1af3726a3c43bb9d3533b2a942959cdf03f04fd0a8e6afd85cc650cfc" Jul 19 08:14:50.552597 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:50.552574 2574 generic.go:356] "Generic (PLEG): container finished" podID="21210c5f-f1fd-4853-85dc-258f9b910be5" containerID="acef55cb73d6c0ebe6d0258c6d1657d721b7c1f790a3e3cce7f56c8a9d4d76f5" exitCode=0 Jul 19 08:14:50.552722 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:50.552685 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-1c28d394-gw-openshift-default-6d59c559cf-w9pht" Jul 19 08:14:50.562428 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:50.562394 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-1c28d394-gw-openshift-default-6d59c559cf-w9pht" event={"ID":"21210c5f-f1fd-4853-85dc-258f9b910be5","Type":"ContainerDied","Data":"acef55cb73d6c0ebe6d0258c6d1657d721b7c1f790a3e3cce7f56c8a9d4d76f5"} Jul 19 08:14:50.562587 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:50.562455 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-1c28d394-gw-openshift-default-6d59c559cf-w9pht" event={"ID":"21210c5f-f1fd-4853-85dc-258f9b910be5","Type":"ContainerDied","Data":"d60ee9dfe3db05b4d0c5c82c2bf4f839e5a46932558400a674cf1efb66877ebf"} Jul 19 08:14:50.564142 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:50.563668 2574 scope.go:117] "RemoveContainer" containerID="acef55cb73d6c0ebe6d0258c6d1657d721b7c1f790a3e3cce7f56c8a9d4d76f5" Jul 19 08:14:50.574047 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:50.574029 2574 scope.go:117] "RemoveContainer" containerID="34da24aeb523485e34da6562a4efd956abefc6fa3a4431886eb9c02a66fcaf36" Jul 19 08:14:50.582176 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:50.582158 2574 scope.go:117] "RemoveContainer" containerID="acef55cb73d6c0ebe6d0258c6d1657d721b7c1f790a3e3cce7f56c8a9d4d76f5" Jul 19 08:14:50.582431 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:14:50.582414 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acef55cb73d6c0ebe6d0258c6d1657d721b7c1f790a3e3cce7f56c8a9d4d76f5\": container with ID starting with acef55cb73d6c0ebe6d0258c6d1657d721b7c1f790a3e3cce7f56c8a9d4d76f5 not found: ID does not exist" containerID="acef55cb73d6c0ebe6d0258c6d1657d721b7c1f790a3e3cce7f56c8a9d4d76f5" Jul 19 08:14:50.582506 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:50.582444 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acef55cb73d6c0ebe6d0258c6d1657d721b7c1f790a3e3cce7f56c8a9d4d76f5"} err="failed to get container status \"acef55cb73d6c0ebe6d0258c6d1657d721b7c1f790a3e3cce7f56c8a9d4d76f5\": rpc error: code = NotFound desc = could not find container \"acef55cb73d6c0ebe6d0258c6d1657d721b7c1f790a3e3cce7f56c8a9d4d76f5\": container with ID starting with acef55cb73d6c0ebe6d0258c6d1657d721b7c1f790a3e3cce7f56c8a9d4d76f5 not found: ID does not exist" Jul 19 08:14:50.582506 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:50.582470 2574 scope.go:117] "RemoveContainer" containerID="34da24aeb523485e34da6562a4efd956abefc6fa3a4431886eb9c02a66fcaf36" Jul 19 08:14:50.582506 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:50.582484 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-1c28d394-gw-openshift-default-6d59c559cf-w9pht"] Jul 19 08:14:50.582731 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:14:50.582714 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34da24aeb523485e34da6562a4efd956abefc6fa3a4431886eb9c02a66fcaf36\": container with ID starting with 34da24aeb523485e34da6562a4efd956abefc6fa3a4431886eb9c02a66fcaf36 not found: ID does not exist" containerID="34da24aeb523485e34da6562a4efd956abefc6fa3a4431886eb9c02a66fcaf36" Jul 19 08:14:50.582774 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:50.582736 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34da24aeb523485e34da6562a4efd956abefc6fa3a4431886eb9c02a66fcaf36"} err="failed to get container status \"34da24aeb523485e34da6562a4efd956abefc6fa3a4431886eb9c02a66fcaf36\": rpc error: code = NotFound desc = could not find container \"34da24aeb523485e34da6562a4efd956abefc6fa3a4431886eb9c02a66fcaf36\": container with ID starting with 34da24aeb523485e34da6562a4efd956abefc6fa3a4431886eb9c02a66fcaf36 not found: ID does not exist" Jul 19 08:14:50.586103 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:50.586081 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-1c28d394-gw-openshift-default-6d59c559cf-w9pht"] Jul 19 08:14:50.597697 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:50.597672 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-1c28d394-767864b5dd-lzkfv"] Jul 19 08:14:50.600941 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:50.600917 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-1c28d394-767864b5dd-lzkfv"] Jul 19 08:14:51.189104 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:51.189069 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:14:52.565928 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:52.565884 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1566fae3-6df6-4396-b8f9-78c96483c6a1" path="/var/lib/kubelet/pods/1566fae3-6df6-4396-b8f9-78c96483c6a1/volumes" Jul 19 08:14:52.566575 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:52.566554 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21210c5f-f1fd-4853-85dc-258f9b910be5" path="/var/lib/kubelet/pods/21210c5f-f1fd-4853-85dc-258f9b910be5/volumes" Jul 19 08:14:56.176975 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:56.176942 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:14:56.280554 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:56.280495 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:14:56.366719 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:56.366675 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-2a60bf57-openshift-default-5bdb955d89-sclw2"] Jul 19 08:14:56.367100 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:56.367072 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1566fae3-6df6-4396-b8f9-78c96483c6a1" containerName="maas-api" Jul 19 08:14:56.367100 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:56.367095 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="1566fae3-6df6-4396-b8f9-78c96483c6a1" containerName="maas-api" Jul 19 08:14:56.367100 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:56.367106 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d813d49-30ab-434d-a246-9734ab3cf5fb" containerName="revoke-keys" Jul 19 08:14:56.367342 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:56.367112 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d813d49-30ab-434d-a246-9734ab3cf5fb" containerName="revoke-keys" Jul 19 08:14:56.367342 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:56.367122 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21210c5f-f1fd-4853-85dc-258f9b910be5" containerName="istio-proxy" Jul 19 08:14:56.367342 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:56.367127 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="21210c5f-f1fd-4853-85dc-258f9b910be5" containerName="istio-proxy" Jul 19 08:14:56.367342 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:56.367137 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1566fae3-6df6-4396-b8f9-78c96483c6a1" containerName="maas-api" Jul 19 08:14:56.367342 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:56.367145 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="1566fae3-6df6-4396-b8f9-78c96483c6a1" containerName="maas-api" Jul 19 08:14:56.367342 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:56.367164 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21210c5f-f1fd-4853-85dc-258f9b910be5" containerName="istio-proxy" Jul 19 08:14:56.367342 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:56.367172 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="21210c5f-f1fd-4853-85dc-258f9b910be5" containerName="istio-proxy" Jul 19 08:14:56.367342 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:56.367192 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d813d49-30ab-434d-a246-9734ab3cf5fb" containerName="revoke-keys" Jul 19 08:14:56.367342 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:56.367200 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d813d49-30ab-434d-a246-9734ab3cf5fb" containerName="revoke-keys" Jul 19 08:14:56.367342 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:56.367213 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d813d49-30ab-434d-a246-9734ab3cf5fb" containerName="revoke-keys" Jul 19 08:14:56.367342 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:56.367221 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d813d49-30ab-434d-a246-9734ab3cf5fb" containerName="revoke-keys" Jul 19 08:14:56.367342 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:56.367321 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d813d49-30ab-434d-a246-9734ab3cf5fb" containerName="revoke-keys" Jul 19 08:14:56.367342 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:56.367333 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="21210c5f-f1fd-4853-85dc-258f9b910be5" containerName="istio-proxy" Jul 19 08:14:56.367342 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:56.367343 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="1566fae3-6df6-4396-b8f9-78c96483c6a1" containerName="maas-api" Jul 19 08:14:56.368004 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:56.367352 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="1566fae3-6df6-4396-b8f9-78c96483c6a1" containerName="maas-api" Jul 19 08:14:56.368004 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:56.367364 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="1566fae3-6df6-4396-b8f9-78c96483c6a1" containerName="maas-api" Jul 19 08:14:56.368004 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:56.367375 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d813d49-30ab-434d-a246-9734ab3cf5fb" containerName="revoke-keys" Jul 19 08:14:56.372421 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:56.372397 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2a60bf57-openshift-default-5bdb955d89-sclw2" Jul 19 08:14:56.374923 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:56.374902 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 19 08:14:56.375043 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:56.374988 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-2a60bf57-openshift-default-dockercfg-8md66\"" Jul 19 08:14:56.382054 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:56.382031 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-2a60bf57-openshift-default-5bdb955d89-sclw2"] Jul 19 08:14:56.397442 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:56.397401 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:14:56.421574 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:56.421539 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxsvm\" (UniqueName: \"kubernetes.io/projected/a81a2508-608d-43ea-89fa-905c6539f8f1-kube-api-access-fxsvm\") pod \"e2e-ait-2a60bf57-openshift-default-5bdb955d89-sclw2\" (UID: \"a81a2508-608d-43ea-89fa-905c6539f8f1\") " pod="openshift-ingress/e2e-ait-2a60bf57-openshift-default-5bdb955d89-sclw2" Jul 19 08:14:56.421718 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:56.421592 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a81a2508-608d-43ea-89fa-905c6539f8f1-credential-socket\") pod \"e2e-ait-2a60bf57-openshift-default-5bdb955d89-sclw2\" (UID: \"a81a2508-608d-43ea-89fa-905c6539f8f1\") " pod="openshift-ingress/e2e-ait-2a60bf57-openshift-default-5bdb955d89-sclw2" Jul 19 08:14:56.421718 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:56.421632 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a81a2508-608d-43ea-89fa-905c6539f8f1-istio-token\") pod \"e2e-ait-2a60bf57-openshift-default-5bdb955d89-sclw2\" (UID: \"a81a2508-608d-43ea-89fa-905c6539f8f1\") " pod="openshift-ingress/e2e-ait-2a60bf57-openshift-default-5bdb955d89-sclw2" Jul 19 08:14:56.421718 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:56.421662 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a81a2508-608d-43ea-89fa-905c6539f8f1-istio-data\") pod \"e2e-ait-2a60bf57-openshift-default-5bdb955d89-sclw2\" (UID: \"a81a2508-608d-43ea-89fa-905c6539f8f1\") " pod="openshift-ingress/e2e-ait-2a60bf57-openshift-default-5bdb955d89-sclw2" Jul 19 08:14:56.421859 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:56.421731 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a81a2508-608d-43ea-89fa-905c6539f8f1-workload-certs\") pod \"e2e-ait-2a60bf57-openshift-default-5bdb955d89-sclw2\" (UID: \"a81a2508-608d-43ea-89fa-905c6539f8f1\") " pod="openshift-ingress/e2e-ait-2a60bf57-openshift-default-5bdb955d89-sclw2" Jul 19 08:14:56.421859 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:56.421795 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a81a2508-608d-43ea-89fa-905c6539f8f1-istio-envoy\") pod \"e2e-ait-2a60bf57-openshift-default-5bdb955d89-sclw2\" (UID: \"a81a2508-608d-43ea-89fa-905c6539f8f1\") " pod="openshift-ingress/e2e-ait-2a60bf57-openshift-default-5bdb955d89-sclw2" Jul 19 08:14:56.421859 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:56.421828 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a81a2508-608d-43ea-89fa-905c6539f8f1-istiod-ca-cert\") pod \"e2e-ait-2a60bf57-openshift-default-5bdb955d89-sclw2\" (UID: \"a81a2508-608d-43ea-89fa-905c6539f8f1\") " pod="openshift-ingress/e2e-ait-2a60bf57-openshift-default-5bdb955d89-sclw2" Jul 19 08:14:56.421991 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:56.421907 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a81a2508-608d-43ea-89fa-905c6539f8f1-workload-socket\") pod \"e2e-ait-2a60bf57-openshift-default-5bdb955d89-sclw2\" (UID: \"a81a2508-608d-43ea-89fa-905c6539f8f1\") " pod="openshift-ingress/e2e-ait-2a60bf57-openshift-default-5bdb955d89-sclw2" Jul 19 08:14:56.421991 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:56.421932 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a81a2508-608d-43ea-89fa-905c6539f8f1-istio-podinfo\") pod \"e2e-ait-2a60bf57-openshift-default-5bdb955d89-sclw2\" (UID: \"a81a2508-608d-43ea-89fa-905c6539f8f1\") " pod="openshift-ingress/e2e-ait-2a60bf57-openshift-default-5bdb955d89-sclw2" Jul 19 08:14:56.492826 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:56.492761 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:14:56.523030 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:56.522998 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a81a2508-608d-43ea-89fa-905c6539f8f1-istio-token\") pod \"e2e-ait-2a60bf57-openshift-default-5bdb955d89-sclw2\" (UID: \"a81a2508-608d-43ea-89fa-905c6539f8f1\") " pod="openshift-ingress/e2e-ait-2a60bf57-openshift-default-5bdb955d89-sclw2" Jul 19 08:14:56.523232 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:56.523041 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a81a2508-608d-43ea-89fa-905c6539f8f1-istio-data\") pod \"e2e-ait-2a60bf57-openshift-default-5bdb955d89-sclw2\" (UID: \"a81a2508-608d-43ea-89fa-905c6539f8f1\") " pod="openshift-ingress/e2e-ait-2a60bf57-openshift-default-5bdb955d89-sclw2" Jul 19 08:14:56.523298 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:56.523276 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a81a2508-608d-43ea-89fa-905c6539f8f1-workload-certs\") pod \"e2e-ait-2a60bf57-openshift-default-5bdb955d89-sclw2\" (UID: \"a81a2508-608d-43ea-89fa-905c6539f8f1\") " pod="openshift-ingress/e2e-ait-2a60bf57-openshift-default-5bdb955d89-sclw2" Jul 19 08:14:56.523551 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:56.523529 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a81a2508-608d-43ea-89fa-905c6539f8f1-istio-data\") pod \"e2e-ait-2a60bf57-openshift-default-5bdb955d89-sclw2\" (UID: \"a81a2508-608d-43ea-89fa-905c6539f8f1\") " pod="openshift-ingress/e2e-ait-2a60bf57-openshift-default-5bdb955d89-sclw2" Jul 19 08:14:56.523644 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:56.523566 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a81a2508-608d-43ea-89fa-905c6539f8f1-workload-certs\") pod \"e2e-ait-2a60bf57-openshift-default-5bdb955d89-sclw2\" (UID: \"a81a2508-608d-43ea-89fa-905c6539f8f1\") " pod="openshift-ingress/e2e-ait-2a60bf57-openshift-default-5bdb955d89-sclw2" Jul 19 08:14:56.523644 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:56.523622 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a81a2508-608d-43ea-89fa-905c6539f8f1-istio-envoy\") pod \"e2e-ait-2a60bf57-openshift-default-5bdb955d89-sclw2\" (UID: \"a81a2508-608d-43ea-89fa-905c6539f8f1\") " pod="openshift-ingress/e2e-ait-2a60bf57-openshift-default-5bdb955d89-sclw2" Jul 19 08:14:56.523756 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:56.523649 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a81a2508-608d-43ea-89fa-905c6539f8f1-istiod-ca-cert\") pod \"e2e-ait-2a60bf57-openshift-default-5bdb955d89-sclw2\" (UID: \"a81a2508-608d-43ea-89fa-905c6539f8f1\") " pod="openshift-ingress/e2e-ait-2a60bf57-openshift-default-5bdb955d89-sclw2" Jul 19 08:14:56.523756 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:56.523734 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a81a2508-608d-43ea-89fa-905c6539f8f1-workload-socket\") pod \"e2e-ait-2a60bf57-openshift-default-5bdb955d89-sclw2\" (UID: \"a81a2508-608d-43ea-89fa-905c6539f8f1\") " pod="openshift-ingress/e2e-ait-2a60bf57-openshift-default-5bdb955d89-sclw2" Jul 19 08:14:56.523863 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:56.523760 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a81a2508-608d-43ea-89fa-905c6539f8f1-istio-podinfo\") pod \"e2e-ait-2a60bf57-openshift-default-5bdb955d89-sclw2\" (UID: \"a81a2508-608d-43ea-89fa-905c6539f8f1\") " pod="openshift-ingress/e2e-ait-2a60bf57-openshift-default-5bdb955d89-sclw2" Jul 19 08:14:56.523863 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:56.523807 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxsvm\" (UniqueName: \"kubernetes.io/projected/a81a2508-608d-43ea-89fa-905c6539f8f1-kube-api-access-fxsvm\") pod \"e2e-ait-2a60bf57-openshift-default-5bdb955d89-sclw2\" (UID: \"a81a2508-608d-43ea-89fa-905c6539f8f1\") " pod="openshift-ingress/e2e-ait-2a60bf57-openshift-default-5bdb955d89-sclw2" Jul 19 08:14:56.523863 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:56.523836 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a81a2508-608d-43ea-89fa-905c6539f8f1-credential-socket\") pod \"e2e-ait-2a60bf57-openshift-default-5bdb955d89-sclw2\" (UID: \"a81a2508-608d-43ea-89fa-905c6539f8f1\") " pod="openshift-ingress/e2e-ait-2a60bf57-openshift-default-5bdb955d89-sclw2" Jul 19 08:14:56.524353 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:56.524329 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a81a2508-608d-43ea-89fa-905c6539f8f1-workload-socket\") pod \"e2e-ait-2a60bf57-openshift-default-5bdb955d89-sclw2\" (UID: \"a81a2508-608d-43ea-89fa-905c6539f8f1\") " pod="openshift-ingress/e2e-ait-2a60bf57-openshift-default-5bdb955d89-sclw2" Jul 19 08:14:56.524816 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:56.524790 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a81a2508-608d-43ea-89fa-905c6539f8f1-credential-socket\") pod \"e2e-ait-2a60bf57-openshift-default-5bdb955d89-sclw2\" (UID: \"a81a2508-608d-43ea-89fa-905c6539f8f1\") " pod="openshift-ingress/e2e-ait-2a60bf57-openshift-default-5bdb955d89-sclw2" Jul 19 08:14:56.524927 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:56.524881 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a81a2508-608d-43ea-89fa-905c6539f8f1-istiod-ca-cert\") pod \"e2e-ait-2a60bf57-openshift-default-5bdb955d89-sclw2\" (UID: \"a81a2508-608d-43ea-89fa-905c6539f8f1\") " pod="openshift-ingress/e2e-ait-2a60bf57-openshift-default-5bdb955d89-sclw2" Jul 19 08:14:56.526093 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:56.526069 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a81a2508-608d-43ea-89fa-905c6539f8f1-istio-envoy\") pod \"e2e-ait-2a60bf57-openshift-default-5bdb955d89-sclw2\" (UID: \"a81a2508-608d-43ea-89fa-905c6539f8f1\") " pod="openshift-ingress/e2e-ait-2a60bf57-openshift-default-5bdb955d89-sclw2" Jul 19 08:14:56.527127 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:56.527098 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a81a2508-608d-43ea-89fa-905c6539f8f1-istio-podinfo\") pod \"e2e-ait-2a60bf57-openshift-default-5bdb955d89-sclw2\" (UID: \"a81a2508-608d-43ea-89fa-905c6539f8f1\") " pod="openshift-ingress/e2e-ait-2a60bf57-openshift-default-5bdb955d89-sclw2" Jul 19 08:14:56.531451 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:56.531425 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a81a2508-608d-43ea-89fa-905c6539f8f1-istio-token\") pod \"e2e-ait-2a60bf57-openshift-default-5bdb955d89-sclw2\" (UID: \"a81a2508-608d-43ea-89fa-905c6539f8f1\") " pod="openshift-ingress/e2e-ait-2a60bf57-openshift-default-5bdb955d89-sclw2" Jul 19 08:14:56.532281 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:56.532250 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxsvm\" (UniqueName: \"kubernetes.io/projected/a81a2508-608d-43ea-89fa-905c6539f8f1-kube-api-access-fxsvm\") pod \"e2e-ait-2a60bf57-openshift-default-5bdb955d89-sclw2\" (UID: \"a81a2508-608d-43ea-89fa-905c6539f8f1\") " pod="openshift-ingress/e2e-ait-2a60bf57-openshift-default-5bdb955d89-sclw2" Jul 19 08:14:56.685281 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:56.685243 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2a60bf57-openshift-default-5bdb955d89-sclw2" Jul 19 08:14:56.754121 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:56.754035 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-v298j"] Jul 19 08:14:56.754677 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:56.754652 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1566fae3-6df6-4396-b8f9-78c96483c6a1" containerName="maas-api" Jul 19 08:14:56.754677 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:56.754677 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="1566fae3-6df6-4396-b8f9-78c96483c6a1" containerName="maas-api" Jul 19 08:14:56.754861 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:56.754783 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="1566fae3-6df6-4396-b8f9-78c96483c6a1" containerName="maas-api" Jul 19 08:14:56.754861 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:56.754803 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d813d49-30ab-434d-a246-9734ab3cf5fb" containerName="revoke-keys" Jul 19 08:14:56.754861 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:56.754816 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="21210c5f-f1fd-4853-85dc-258f9b910be5" containerName="istio-proxy" Jul 19 08:14:56.759463 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:56.759436 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-v298j" Jul 19 08:14:56.762162 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:56.761921 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qbdd4\"" Jul 19 08:14:56.762162 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:56.762143 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 19 08:14:56.764093 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:56.764052 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-v298j"] Jul 19 08:14:56.827233 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:56.827195 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d21b86e6-517d-44f3-9422-5ba22f78afe8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef124b28-v298j\" (UID: \"d21b86e6-517d-44f3-9422-5ba22f78afe8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-v298j" Jul 19 08:14:56.834471 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:56.834447 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-2a60bf57-openshift-default-5bdb955d89-sclw2"] Jul 19 08:14:56.836585 ip-10-0-136-70 kubenswrapper[2574]: W0719 08:14:56.836556 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda81a2508_608d_43ea_89fa_905c6539f8f1.slice/crio-354ec0ae7a371184cc41cf50ebc5572d1194a6096e156d8043167daa0563dafb WatchSource:0}: Error finding container 354ec0ae7a371184cc41cf50ebc5572d1194a6096e156d8043167daa0563dafb: Status 404 returned error can't find the container with id 354ec0ae7a371184cc41cf50ebc5572d1194a6096e156d8043167daa0563dafb Jul 19 08:14:56.839117 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:56.839084 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 19 08:14:56.839186 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:56.839150 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 19 08:14:56.839230 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:56.839187 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 19 08:14:56.928725 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:56.928684 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d21b86e6-517d-44f3-9422-5ba22f78afe8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef124b28-v298j\" (UID: \"d21b86e6-517d-44f3-9422-5ba22f78afe8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-v298j" Jul 19 08:14:56.929280 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:56.929261 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d21b86e6-517d-44f3-9422-5ba22f78afe8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef124b28-v298j\" (UID: \"d21b86e6-517d-44f3-9422-5ba22f78afe8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-v298j" Jul 19 08:14:57.074484 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:57.074389 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-v298j" Jul 19 08:14:57.216689 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:57.216665 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-v298j"] Jul 19 08:14:57.220155 ip-10-0-136-70 kubenswrapper[2574]: W0719 08:14:57.220124 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd21b86e6_517d_44f3_9422_5ba22f78afe8.slice/crio-06c9caa4cbb7bf53e019a55079ca3804e2b924dec4476f798d3ff892688dbcc9 WatchSource:0}: Error finding container 06c9caa4cbb7bf53e019a55079ca3804e2b924dec4476f798d3ff892688dbcc9: Status 404 returned error can't find the container with id 06c9caa4cbb7bf53e019a55079ca3804e2b924dec4476f798d3ff892688dbcc9 Jul 19 08:14:57.504339 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:57.504285 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:14:57.591814 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:57.591776 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-v298j" event={"ID":"d21b86e6-517d-44f3-9422-5ba22f78afe8","Type":"ContainerStarted","Data":"182c7c276bd0f6ab8d876874fd28d03b9afc95fc6cc6498e2b8e714a5a03831f"} Jul 19 08:14:57.592039 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:57.591830 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-v298j" event={"ID":"d21b86e6-517d-44f3-9422-5ba22f78afe8","Type":"ContainerStarted","Data":"06c9caa4cbb7bf53e019a55079ca3804e2b924dec4476f798d3ff892688dbcc9"} Jul 19 08:14:57.593435 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:57.593408 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2a60bf57-openshift-default-5bdb955d89-sclw2" event={"ID":"a81a2508-608d-43ea-89fa-905c6539f8f1","Type":"ContainerStarted","Data":"16b27542354d40a618de6705149c4c03ae7220f6f6cd59254b36f89bd830af78"} Jul 19 08:14:57.593576 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:57.593458 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2a60bf57-openshift-default-5bdb955d89-sclw2" event={"ID":"a81a2508-608d-43ea-89fa-905c6539f8f1","Type":"ContainerStarted","Data":"354ec0ae7a371184cc41cf50ebc5572d1194a6096e156d8043167daa0563dafb"} Jul 19 08:14:57.607092 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:57.607044 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-v298j" podStartSLOduration=1.6070279699999999 podStartE2EDuration="1.60702797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 08:14:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:14:57.605758131 +0000 UTC m=+1963.960067732" watchObservedRunningTime="2026-07-19 08:14:57.60702797 +0000 UTC m=+1963.961337548" Jul 19 08:14:57.613871 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:57.613838 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:14:57.633919 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:57.633871 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-2a60bf57-openshift-default-5bdb955d89-sclw2" podStartSLOduration=1.6338551049999999 podStartE2EDuration="1.633855105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 08:14:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:14:57.631694128 +0000 UTC m=+1963.986003729" watchObservedRunningTime="2026-07-19 08:14:57.633855105 +0000 UTC m=+1963.988164706" Jul 19 08:14:57.637250 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:57.637222 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq"] Jul 19 08:14:57.637911 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:57.637745 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1566fae3-6df6-4396-b8f9-78c96483c6a1" containerName="maas-api" Jul 19 08:14:57.637911 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:57.637771 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="1566fae3-6df6-4396-b8f9-78c96483c6a1" containerName="maas-api" Jul 19 08:14:57.641198 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:57.641172 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq" Jul 19 08:14:57.644389 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:57.644365 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-tdbkh\"" Jul 19 08:14:57.644687 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:57.644665 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-2a60bf57-serving-cert\"" Jul 19 08:14:57.645220 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:57.645183 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 19 08:14:57.662195 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:57.662167 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq"] Jul 19 08:14:57.685639 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:57.685559 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-2a60bf57-openshift-default-5bdb955d89-sclw2" Jul 19 08:14:57.690261 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:57.690237 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-2a60bf57-openshift-default-5bdb955d89-sclw2" Jul 19 08:14:57.707508 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:57.707481 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:14:57.735885 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:57.735856 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89d37cf6-8273-433a-8a54-a94ae3059a6d-maas-api-tls\") pod \"maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq\" (UID: \"89d37cf6-8273-433a-8a54-a94ae3059a6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq" Jul 19 08:14:57.736053 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:57.735939 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89d37cf6-8273-433a-8a54-a94ae3059a6d-openshift-service-ca\") pod \"maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq\" (UID: \"89d37cf6-8273-433a-8a54-a94ae3059a6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq" Jul 19 08:14:57.736053 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:57.735991 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmm9t\" (UniqueName: \"kubernetes.io/projected/89d37cf6-8273-433a-8a54-a94ae3059a6d-kube-api-access-lmm9t\") pod \"maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq\" (UID: \"89d37cf6-8273-433a-8a54-a94ae3059a6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq" Jul 19 08:14:57.806060 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:57.806030 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:14:57.836491 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:57.836450 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmm9t\" (UniqueName: \"kubernetes.io/projected/89d37cf6-8273-433a-8a54-a94ae3059a6d-kube-api-access-lmm9t\") pod \"maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq\" (UID: \"89d37cf6-8273-433a-8a54-a94ae3059a6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq" Jul 19 08:14:57.836704 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:57.836538 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89d37cf6-8273-433a-8a54-a94ae3059a6d-maas-api-tls\") pod \"maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq\" (UID: \"89d37cf6-8273-433a-8a54-a94ae3059a6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq" Jul 19 08:14:57.836704 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:57.836596 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89d37cf6-8273-433a-8a54-a94ae3059a6d-openshift-service-ca\") pod \"maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq\" (UID: \"89d37cf6-8273-433a-8a54-a94ae3059a6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq" Jul 19 08:14:57.837225 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:57.837203 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89d37cf6-8273-433a-8a54-a94ae3059a6d-openshift-service-ca\") pod \"maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq\" (UID: \"89d37cf6-8273-433a-8a54-a94ae3059a6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq" Jul 19 08:14:57.839054 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:57.839027 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89d37cf6-8273-433a-8a54-a94ae3059a6d-maas-api-tls\") pod \"maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq\" (UID: \"89d37cf6-8273-433a-8a54-a94ae3059a6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq" Jul 19 08:14:57.845384 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:57.845353 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmm9t\" (UniqueName: \"kubernetes.io/projected/89d37cf6-8273-433a-8a54-a94ae3059a6d-kube-api-access-lmm9t\") pod \"maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq\" (UID: \"89d37cf6-8273-433a-8a54-a94ae3059a6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq" Jul 19 08:14:57.962675 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:57.962645 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq" Jul 19 08:14:58.106259 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:58.106232 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq"] Jul 19 08:14:58.108547 ip-10-0-136-70 kubenswrapper[2574]: W0719 08:14:58.108499 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89d37cf6_8273_433a_8a54_a94ae3059a6d.slice/crio-501266eb5dac8680d3fdb16d26ae805ca02593a2d5f1b9d40a89f9f1a0a95ce7 WatchSource:0}: Error finding container 501266eb5dac8680d3fdb16d26ae805ca02593a2d5f1b9d40a89f9f1a0a95ce7: Status 404 returned error can't find the container with id 501266eb5dac8680d3fdb16d26ae805ca02593a2d5f1b9d40a89f9f1a0a95ce7 Jul 19 08:14:58.275542 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:58.275432 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:14:58.376091 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:58.376058 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:14:58.600389 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:58.600362 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq" event={"ID":"89d37cf6-8273-433a-8a54-a94ae3059a6d","Type":"ContainerStarted","Data":"501266eb5dac8680d3fdb16d26ae805ca02593a2d5f1b9d40a89f9f1a0a95ce7"} Jul 19 08:14:58.602131 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:58.602104 2574 generic.go:356] "Generic (PLEG): container finished" podID="d21b86e6-517d-44f3-9422-5ba22f78afe8" containerID="182c7c276bd0f6ab8d876874fd28d03b9afc95fc6cc6498e2b8e714a5a03831f" exitCode=6 Jul 19 08:14:58.602245 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:58.602181 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-v298j" event={"ID":"d21b86e6-517d-44f3-9422-5ba22f78afe8","Type":"ContainerDied","Data":"182c7c276bd0f6ab8d876874fd28d03b9afc95fc6cc6498e2b8e714a5a03831f"} Jul 19 08:14:58.602387 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:58.602373 2574 scope.go:117] "RemoveContainer" containerID="182c7c276bd0f6ab8d876874fd28d03b9afc95fc6cc6498e2b8e714a5a03831f" Jul 19 08:14:58.602574 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:58.602552 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-2a60bf57-openshift-default-5bdb955d89-sclw2" Jul 19 08:14:58.603617 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:58.603600 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-2a60bf57-openshift-default-5bdb955d89-sclw2" Jul 19 08:14:59.609070 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:59.608974 2574 generic.go:356] "Generic (PLEG): container finished" podID="89d37cf6-8273-433a-8a54-a94ae3059a6d" containerID="c43bd625d983ae45b2f9a5fe98900ffee7d2e4876c8c21398ecdf9998d5e3e4e" exitCode=1 Jul 19 08:14:59.609070 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:59.609059 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq" event={"ID":"89d37cf6-8273-433a-8a54-a94ae3059a6d","Type":"ContainerDied","Data":"c43bd625d983ae45b2f9a5fe98900ffee7d2e4876c8c21398ecdf9998d5e3e4e"} Jul 19 08:14:59.609568 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:59.609291 2574 scope.go:117] "RemoveContainer" containerID="c43bd625d983ae45b2f9a5fe98900ffee7d2e4876c8c21398ecdf9998d5e3e4e" Jul 19 08:14:59.611092 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:59.611069 2574 generic.go:356] "Generic (PLEG): container finished" podID="d21b86e6-517d-44f3-9422-5ba22f78afe8" containerID="0324d493349be3204d172e9b096f72ddbee4f149ebc33dd0912e78b73df73ebf" exitCode=6 Jul 19 08:14:59.611206 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:59.611158 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-v298j" event={"ID":"d21b86e6-517d-44f3-9422-5ba22f78afe8","Type":"ContainerDied","Data":"0324d493349be3204d172e9b096f72ddbee4f149ebc33dd0912e78b73df73ebf"} Jul 19 08:14:59.611245 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:59.611216 2574 scope.go:117] "RemoveContainer" containerID="182c7c276bd0f6ab8d876874fd28d03b9afc95fc6cc6498e2b8e714a5a03831f" Jul 19 08:14:59.611329 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:59.611316 2574 scope.go:117] "RemoveContainer" containerID="0324d493349be3204d172e9b096f72ddbee4f149ebc33dd0912e78b73df73ebf" Jul 19 08:14:59.611594 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:14:59.611500 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ef124b28-v298j_odh-ai-gateway-infra(d21b86e6-517d-44f3-9422-5ba22f78afe8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-v298j" podUID="d21b86e6-517d-44f3-9422-5ba22f78afe8" Jul 19 08:14:59.809892 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:14:59.809856 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:15:00.618586 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:15:00.618465 2574 generic.go:356] "Generic (PLEG): container finished" podID="89d37cf6-8273-433a-8a54-a94ae3059a6d" containerID="72b216f6f6166e068eb0e66c1f7b1d300bd99dd1db87069d20d29dde15740e8b" exitCode=1 Jul 19 08:15:00.618586 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:15:00.618558 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq" event={"ID":"89d37cf6-8273-433a-8a54-a94ae3059a6d","Type":"ContainerDied","Data":"72b216f6f6166e068eb0e66c1f7b1d300bd99dd1db87069d20d29dde15740e8b"} Jul 19 08:15:00.619113 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:15:00.618627 2574 scope.go:117] "RemoveContainer" containerID="c43bd625d983ae45b2f9a5fe98900ffee7d2e4876c8c21398ecdf9998d5e3e4e" Jul 19 08:15:00.619113 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:15:00.618874 2574 scope.go:117] "RemoveContainer" containerID="72b216f6f6166e068eb0e66c1f7b1d300bd99dd1db87069d20d29dde15740e8b" Jul 19 08:15:00.619216 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:15:00.619105 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq_odh-ai-gateway-infra(89d37cf6-8273-433a-8a54-a94ae3059a6d)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq" podUID="89d37cf6-8273-433a-8a54-a94ae3059a6d" Jul 19 08:15:00.620684 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:15:00.620663 2574 scope.go:117] "RemoveContainer" containerID="0324d493349be3204d172e9b096f72ddbee4f149ebc33dd0912e78b73df73ebf" Jul 19 08:15:00.620902 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:15:00.620879 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ef124b28-v298j_odh-ai-gateway-infra(d21b86e6-517d-44f3-9422-5ba22f78afe8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-v298j" podUID="d21b86e6-517d-44f3-9422-5ba22f78afe8" Jul 19 08:15:01.474197 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:15:01.474160 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 08:15:01.629950 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:15:01.629918 2574 scope.go:117] "RemoveContainer" containerID="72b216f6f6166e068eb0e66c1f7b1d300bd99dd1db87069d20d29dde15740e8b" Jul 19 08:15:01.630330 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:15:01.630092 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq_odh-ai-gateway-infra(89d37cf6-8273-433a-8a54-a94ae3059a6d)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq" podUID="89d37cf6-8273-433a-8a54-a94ae3059a6d" Jul 19 08:15:02.963083 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:15:02.963044 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq" Jul 19 08:15:02.963488 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:15:02.963442 2574 scope.go:117] "RemoveContainer" containerID="72b216f6f6166e068eb0e66c1f7b1d300bd99dd1db87069d20d29dde15740e8b" Jul 19 08:15:02.963675 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:15:02.963653 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq_odh-ai-gateway-infra(89d37cf6-8273-433a-8a54-a94ae3059a6d)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq" podUID="89d37cf6-8273-433a-8a54-a94ae3059a6d" Jul 19 08:15:07.962954 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:15:07.962907 2574 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq" Jul 19 08:15:07.963449 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:15:07.963432 2574 scope.go:117] "RemoveContainer" containerID="72b216f6f6166e068eb0e66c1f7b1d300bd99dd1db87069d20d29dde15740e8b" Jul 19 08:15:07.963704 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:15:07.963677 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq_odh-ai-gateway-infra(89d37cf6-8273-433a-8a54-a94ae3059a6d)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq" podUID="89d37cf6-8273-433a-8a54-a94ae3059a6d" Jul 19 08:15:09.682186 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:15:09.682107 2574 generic.go:356] "Generic (PLEG): container finished" podID="a81a2508-608d-43ea-89fa-905c6539f8f1" containerID="16b27542354d40a618de6705149c4c03ae7220f6f6cd59254b36f89bd830af78" exitCode=0 Jul 19 08:15:09.682621 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:15:09.682177 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2a60bf57-openshift-default-5bdb955d89-sclw2" event={"ID":"a81a2508-608d-43ea-89fa-905c6539f8f1","Type":"ContainerDied","Data":"16b27542354d40a618de6705149c4c03ae7220f6f6cd59254b36f89bd830af78"} Jul 19 08:15:09.682621 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:15:09.682611 2574 scope.go:117] "RemoveContainer" containerID="16b27542354d40a618de6705149c4c03ae7220f6f6cd59254b36f89bd830af78" Jul 19 08:15:09.683653 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:15:09.683626 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 19 08:15:09.683744 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:15:09.683696 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 19 08:15:09.683744 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:15:09.683730 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 19 08:15:10.689101 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:15:10.689065 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2a60bf57-openshift-default-5bdb955d89-sclw2" event={"ID":"a81a2508-608d-43ea-89fa-905c6539f8f1","Type":"ContainerStarted","Data":"b1ffbbec0d8a0932d7652a9fd632a57773a1b286cd43b6dee085f301eef917e0"} Jul 19 08:15:11.686214 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:15:11.686177 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-2a60bf57-openshift-default-5bdb955d89-sclw2" Jul 19 08:15:11.687282 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:15:11.687258 2574 patch_prober.go:28] interesting pod/e2e-ait-2a60bf57-openshift-default-5bdb955d89-sclw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Jul 19 08:15:11.687400 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:15:11.687313 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2a60bf57-openshift-default-5bdb955d89-sclw2" podUID="a81a2508-608d-43ea-89fa-905c6539f8f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Jul 19 08:15:12.552751 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:15:12.552723 2574 scope.go:117] "RemoveContainer" containerID="0324d493349be3204d172e9b096f72ddbee4f149ebc33dd0912e78b73df73ebf" Jul 19 08:15:12.685632 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:15:12.685592 2574 patch_prober.go:28] interesting pod/e2e-ait-2a60bf57-openshift-default-5bdb955d89-sclw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Jul 19 08:15:12.685809 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:15:12.685648 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2a60bf57-openshift-default-5bdb955d89-sclw2" podUID="a81a2508-608d-43ea-89fa-905c6539f8f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Jul 19 08:15:13.602565 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:15:13.602504 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-2a60bf57-openshift-default-5bdb955d89-sclw2" Jul 19 08:15:13.685939 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:15:13.685837 2574 patch_prober.go:28] interesting pod/e2e-ait-2a60bf57-openshift-default-5bdb955d89-sclw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Jul 19 08:15:13.685939 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:15:13.685904 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2a60bf57-openshift-default-5bdb955d89-sclw2" podUID="a81a2508-608d-43ea-89fa-905c6539f8f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Jul 19 08:15:13.712770 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:15:13.712735 2574 generic.go:356] "Generic (PLEG): container finished" podID="d21b86e6-517d-44f3-9422-5ba22f78afe8" containerID="2a84013984215f7710546d7f436e58f583b0defd80187212967f8ca25ebb246a" exitCode=6 Jul 19 08:15:13.712968 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:15:13.712783 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-v298j" event={"ID":"d21b86e6-517d-44f3-9422-5ba22f78afe8","Type":"ContainerDied","Data":"2a84013984215f7710546d7f436e58f583b0defd80187212967f8ca25ebb246a"} Jul 19 08:15:13.712968 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:15:13.712844 2574 scope.go:117] "RemoveContainer" containerID="0324d493349be3204d172e9b096f72ddbee4f149ebc33dd0912e78b73df73ebf" Jul 19 08:15:13.713191 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:15:13.713167 2574 scope.go:117] "RemoveContainer" containerID="2a84013984215f7710546d7f436e58f583b0defd80187212967f8ca25ebb246a" Jul 19 08:15:13.713414 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:15:13.713393 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ef124b28-v298j_odh-ai-gateway-infra(d21b86e6-517d-44f3-9422-5ba22f78afe8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-v298j" podUID="d21b86e6-517d-44f3-9422-5ba22f78afe8" Jul 19 08:15:14.731895 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:15:14.731862 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-2a60bf57-openshift-default-5bdb955d89-sclw2" Jul 19 08:15:14.732952 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:15:14.732933 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-2a60bf57-openshift-default-5bdb955d89-sclw2" Jul 19 08:15:14.736862 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:15:14.736833 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-v298j"] Jul 19 08:15:14.920289 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:15:14.920257 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-v298j" Jul 19 08:15:15.008054 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:15:15.007960 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d21b86e6-517d-44f3-9422-5ba22f78afe8-maas-api-service-ca\") pod \"d21b86e6-517d-44f3-9422-5ba22f78afe8\" (UID: \"d21b86e6-517d-44f3-9422-5ba22f78afe8\") " Jul 19 08:15:15.008333 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:15:15.008309 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d21b86e6-517d-44f3-9422-5ba22f78afe8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d21b86e6-517d-44f3-9422-5ba22f78afe8" (UID: "d21b86e6-517d-44f3-9422-5ba22f78afe8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 08:15:15.109171 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:15:15.109136 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d21b86e6-517d-44f3-9422-5ba22f78afe8-maas-api-service-ca\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:15:15.727794 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:15:15.727763 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-v298j" Jul 19 08:15:15.728026 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:15:15.727796 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-v298j" event={"ID":"d21b86e6-517d-44f3-9422-5ba22f78afe8","Type":"ContainerDied","Data":"06c9caa4cbb7bf53e019a55079ca3804e2b924dec4476f798d3ff892688dbcc9"} Jul 19 08:15:15.728026 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:15:15.727855 2574 scope.go:117] "RemoveContainer" containerID="2a84013984215f7710546d7f436e58f583b0defd80187212967f8ca25ebb246a" Jul 19 08:15:15.754631 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:15:15.754602 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-v298j"] Jul 19 08:15:15.758749 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:15:15.758721 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-v298j"] Jul 19 08:15:16.559002 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:15:16.558965 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d21b86e6-517d-44f3-9422-5ba22f78afe8" path="/var/lib/kubelet/pods/d21b86e6-517d-44f3-9422-5ba22f78afe8/volumes" Jul 19 08:15:20.552681 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:15:20.552643 2574 scope.go:117] "RemoveContainer" containerID="72b216f6f6166e068eb0e66c1f7b1d300bd99dd1db87069d20d29dde15740e8b" Jul 19 08:15:21.759535 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:15:21.759487 2574 generic.go:356] "Generic (PLEG): container finished" podID="89d37cf6-8273-433a-8a54-a94ae3059a6d" containerID="9340c684aaa6d913512d0e285ca6bc88c82ef19c7b4c4810c6a030799480b946" exitCode=1 Jul 19 08:15:21.759967 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:15:21.759562 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq" event={"ID":"89d37cf6-8273-433a-8a54-a94ae3059a6d","Type":"ContainerDied","Data":"9340c684aaa6d913512d0e285ca6bc88c82ef19c7b4c4810c6a030799480b946"} Jul 19 08:15:21.759967 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:15:21.759623 2574 scope.go:117] "RemoveContainer" containerID="72b216f6f6166e068eb0e66c1f7b1d300bd99dd1db87069d20d29dde15740e8b" Jul 19 08:15:21.760094 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:15:21.760000 2574 scope.go:117] "RemoveContainer" containerID="9340c684aaa6d913512d0e285ca6bc88c82ef19c7b4c4810c6a030799480b946" Jul 19 08:15:21.760237 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:15:21.760218 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq_odh-ai-gateway-infra(89d37cf6-8273-433a-8a54-a94ae3059a6d)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq" podUID="89d37cf6-8273-433a-8a54-a94ae3059a6d" Jul 19 08:15:22.963470 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:15:22.963426 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq" Jul 19 08:15:22.963947 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:15:22.963844 2574 scope.go:117] "RemoveContainer" containerID="9340c684aaa6d913512d0e285ca6bc88c82ef19c7b4c4810c6a030799480b946" Jul 19 08:15:22.964056 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:15:22.964034 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq_odh-ai-gateway-infra(89d37cf6-8273-433a-8a54-a94ae3059a6d)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq" podUID="89d37cf6-8273-433a-8a54-a94ae3059a6d" Jul 19 08:15:27.963466 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:15:27.963426 2574 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq" Jul 19 08:15:27.963997 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:15:27.963868 2574 scope.go:117] "RemoveContainer" containerID="9340c684aaa6d913512d0e285ca6bc88c82ef19c7b4c4810c6a030799480b946" Jul 19 08:15:27.964078 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:15:27.964056 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq_odh-ai-gateway-infra(89d37cf6-8273-433a-8a54-a94ae3059a6d)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq" podUID="89d37cf6-8273-433a-8a54-a94ae3059a6d" Jul 19 08:15:39.552943 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:15:39.552909 2574 scope.go:117] "RemoveContainer" containerID="9340c684aaa6d913512d0e285ca6bc88c82ef19c7b4c4810c6a030799480b946" Jul 19 08:15:39.553343 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:15:39.553102 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq_odh-ai-gateway-infra(89d37cf6-8273-433a-8a54-a94ae3059a6d)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq" podUID="89d37cf6-8273-433a-8a54-a94ae3059a6d" Jul 19 08:15:50.553111 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:15:50.553075 2574 scope.go:117] "RemoveContainer" containerID="9340c684aaa6d913512d0e285ca6bc88c82ef19c7b4c4810c6a030799480b946" Jul 19 08:15:50.553596 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:15:50.553348 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 19 08:15:51.916174 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:15:51.916136 2574 generic.go:356] "Generic (PLEG): container finished" podID="89d37cf6-8273-433a-8a54-a94ae3059a6d" containerID="2aa539738b6c7969b6180cef219a749053679ed3ff320a0643b410f0cc402a32" exitCode=1 Jul 19 08:15:51.916655 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:15:51.916186 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq" event={"ID":"89d37cf6-8273-433a-8a54-a94ae3059a6d","Type":"ContainerDied","Data":"2aa539738b6c7969b6180cef219a749053679ed3ff320a0643b410f0cc402a32"} Jul 19 08:15:51.916655 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:15:51.916239 2574 scope.go:117] "RemoveContainer" containerID="9340c684aaa6d913512d0e285ca6bc88c82ef19c7b4c4810c6a030799480b946" Jul 19 08:15:51.916754 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:15:51.916650 2574 scope.go:117] "RemoveContainer" containerID="2aa539738b6c7969b6180cef219a749053679ed3ff320a0643b410f0cc402a32" Jul 19 08:15:51.916904 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:15:51.916879 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq_odh-ai-gateway-infra(89d37cf6-8273-433a-8a54-a94ae3059a6d)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq" podUID="89d37cf6-8273-433a-8a54-a94ae3059a6d" Jul 19 08:15:52.963189 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:15:52.963147 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq" Jul 19 08:15:52.963642 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:15:52.963546 2574 scope.go:117] "RemoveContainer" containerID="2aa539738b6c7969b6180cef219a749053679ed3ff320a0643b410f0cc402a32" Jul 19 08:15:52.963772 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:15:52.963751 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq_odh-ai-gateway-infra(89d37cf6-8273-433a-8a54-a94ae3059a6d)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq" podUID="89d37cf6-8273-433a-8a54-a94ae3059a6d" Jul 19 08:15:57.503835 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:15:57.503796 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:15:57.549016 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:15:57.548975 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:15:57.962772 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:15:57.962740 2574 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq" Jul 19 08:15:57.963169 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:15:57.963154 2574 scope.go:117] "RemoveContainer" containerID="2aa539738b6c7969b6180cef219a749053679ed3ff320a0643b410f0cc402a32" Jul 19 08:15:57.963368 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:15:57.963345 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq_odh-ai-gateway-infra(89d37cf6-8273-433a-8a54-a94ae3059a6d)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq" podUID="89d37cf6-8273-433a-8a54-a94ae3059a6d" Jul 19 08:16:11.553502 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:16:11.553467 2574 scope.go:117] "RemoveContainer" containerID="2aa539738b6c7969b6180cef219a749053679ed3ff320a0643b410f0cc402a32" Jul 19 08:16:11.554014 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:16:11.553692 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq_odh-ai-gateway-infra(89d37cf6-8273-433a-8a54-a94ae3059a6d)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq" podUID="89d37cf6-8273-433a-8a54-a94ae3059a6d" Jul 19 08:16:25.552846 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:16:25.552811 2574 scope.go:117] "RemoveContainer" containerID="2aa539738b6c7969b6180cef219a749053679ed3ff320a0643b410f0cc402a32" Jul 19 08:16:25.553348 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:16:25.553008 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq_odh-ai-gateway-infra(89d37cf6-8273-433a-8a54-a94ae3059a6d)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq" podUID="89d37cf6-8273-433a-8a54-a94ae3059a6d" Jul 19 08:16:40.552689 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:16:40.552601 2574 scope.go:117] "RemoveContainer" containerID="2aa539738b6c7969b6180cef219a749053679ed3ff320a0643b410f0cc402a32" Jul 19 08:16:41.165637 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:16:41.165602 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq" event={"ID":"89d37cf6-8273-433a-8a54-a94ae3059a6d","Type":"ContainerStarted","Data":"eb76874aa1b64510012d2f6a8b664bd727ce7bf4c667d52f182c8455bdec0e4b"} Jul 19 08:16:41.165870 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:16:41.165849 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq" Jul 19 08:16:41.182483 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:16:41.182429 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq" podStartSLOduration=102.335187301 podStartE2EDuration="1m44.182414711s" totalImagesPullingTime="1.84722741s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 08:14:57 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:16:41.181091083 +0000 UTC m=+2067.535400680" watchObservedRunningTime="2026-07-19 08:16:41.182414711 +0000 UTC m=+2067.536724311" Jul 19 08:16:42.173029 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:16:42.172993 2574 generic.go:356] "Generic (PLEG): container finished" podID="89d37cf6-8273-433a-8a54-a94ae3059a6d" containerID="eb76874aa1b64510012d2f6a8b664bd727ce7bf4c667d52f182c8455bdec0e4b" exitCode=1 Jul 19 08:16:42.173538 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:16:42.173042 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq" event={"ID":"89d37cf6-8273-433a-8a54-a94ae3059a6d","Type":"ContainerDied","Data":"eb76874aa1b64510012d2f6a8b664bd727ce7bf4c667d52f182c8455bdec0e4b"} Jul 19 08:16:42.173538 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:16:42.173088 2574 scope.go:117] "RemoveContainer" containerID="2aa539738b6c7969b6180cef219a749053679ed3ff320a0643b410f0cc402a32" Jul 19 08:16:42.173538 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:16:42.173358 2574 scope.go:117] "RemoveContainer" containerID="eb76874aa1b64510012d2f6a8b664bd727ce7bf4c667d52f182c8455bdec0e4b" Jul 19 08:16:42.173709 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:16:42.173600 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq_odh-ai-gateway-infra(89d37cf6-8273-433a-8a54-a94ae3059a6d)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq" podUID="89d37cf6-8273-433a-8a54-a94ae3059a6d" Jul 19 08:16:43.180126 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:16:43.180096 2574 scope.go:117] "RemoveContainer" containerID="eb76874aa1b64510012d2f6a8b664bd727ce7bf4c667d52f182c8455bdec0e4b" Jul 19 08:16:43.180554 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:16:43.180261 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq_odh-ai-gateway-infra(89d37cf6-8273-433a-8a54-a94ae3059a6d)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq" podUID="89d37cf6-8273-433a-8a54-a94ae3059a6d" Jul 19 08:16:47.963531 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:16:47.963467 2574 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq" Jul 19 08:16:47.964174 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:16:47.964062 2574 scope.go:117] "RemoveContainer" containerID="eb76874aa1b64510012d2f6a8b664bd727ce7bf4c667d52f182c8455bdec0e4b" Jul 19 08:16:47.964334 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:16:47.964309 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq_odh-ai-gateway-infra(89d37cf6-8273-433a-8a54-a94ae3059a6d)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq" podUID="89d37cf6-8273-433a-8a54-a94ae3059a6d" Jul 19 08:17:00.553220 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:17:00.553185 2574 scope.go:117] "RemoveContainer" containerID="eb76874aa1b64510012d2f6a8b664bd727ce7bf4c667d52f182c8455bdec0e4b" Jul 19 08:17:00.553697 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:17:00.553379 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq_odh-ai-gateway-infra(89d37cf6-8273-433a-8a54-a94ae3059a6d)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq" podUID="89d37cf6-8273-433a-8a54-a94ae3059a6d" Jul 19 08:17:14.662472 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:17:14.662441 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wmhfd_dd68d31d-f763-433a-867b-65b12eabac76/console-operator/1.log" Jul 19 08:17:14.665187 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:17:14.665165 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wmhfd_dd68d31d-f763-433a-867b-65b12eabac76/console-operator/1.log" Jul 19 08:17:15.553429 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:17:15.553397 2574 scope.go:117] "RemoveContainer" containerID="eb76874aa1b64510012d2f6a8b664bd727ce7bf4c667d52f182c8455bdec0e4b" Jul 19 08:17:15.553668 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:17:15.553644 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq_odh-ai-gateway-infra(89d37cf6-8273-433a-8a54-a94ae3059a6d)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq" podUID="89d37cf6-8273-433a-8a54-a94ae3059a6d" Jul 19 08:17:26.552873 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:17:26.552844 2574 scope.go:117] "RemoveContainer" containerID="eb76874aa1b64510012d2f6a8b664bd727ce7bf4c667d52f182c8455bdec0e4b" Jul 19 08:17:26.553336 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:17:26.553042 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq_odh-ai-gateway-infra(89d37cf6-8273-433a-8a54-a94ae3059a6d)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq" podUID="89d37cf6-8273-433a-8a54-a94ae3059a6d" Jul 19 08:17:32.748828 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:17:32.748779 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:17:32.837578 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:17:32.837507 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:17:41.553420 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:17:41.553384 2574 scope.go:117] "RemoveContainer" containerID="eb76874aa1b64510012d2f6a8b664bd727ce7bf4c667d52f182c8455bdec0e4b" Jul 19 08:17:41.553940 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:17:41.553613 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq_odh-ai-gateway-infra(89d37cf6-8273-433a-8a54-a94ae3059a6d)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq" podUID="89d37cf6-8273-433a-8a54-a94ae3059a6d" Jul 19 08:17:55.552995 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:17:55.552959 2574 scope.go:117] "RemoveContainer" containerID="eb76874aa1b64510012d2f6a8b664bd727ce7bf4c667d52f182c8455bdec0e4b" Jul 19 08:17:55.553459 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:17:55.553105 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq_odh-ai-gateway-infra(89d37cf6-8273-433a-8a54-a94ae3059a6d)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq" podUID="89d37cf6-8273-433a-8a54-a94ae3059a6d" Jul 19 08:18:09.553704 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:18:09.553623 2574 scope.go:117] "RemoveContainer" containerID="eb76874aa1b64510012d2f6a8b664bd727ce7bf4c667d52f182c8455bdec0e4b" Jul 19 08:18:10.629744 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:18:10.629711 2574 generic.go:356] "Generic (PLEG): container finished" podID="89d37cf6-8273-433a-8a54-a94ae3059a6d" containerID="8c2e73973e40e2465bb79862e37e8935962b59049a75e7b80e8928a15faedc7c" exitCode=1 Jul 19 08:18:10.630133 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:18:10.629765 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq" event={"ID":"89d37cf6-8273-433a-8a54-a94ae3059a6d","Type":"ContainerDied","Data":"8c2e73973e40e2465bb79862e37e8935962b59049a75e7b80e8928a15faedc7c"} Jul 19 08:18:10.630629 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:18:10.629817 2574 scope.go:117] "RemoveContainer" containerID="eb76874aa1b64510012d2f6a8b664bd727ce7bf4c667d52f182c8455bdec0e4b" Jul 19 08:18:10.630720 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:18:10.630707 2574 scope.go:117] "RemoveContainer" containerID="8c2e73973e40e2465bb79862e37e8935962b59049a75e7b80e8928a15faedc7c" Jul 19 08:18:10.630981 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:18:10.630953 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq_odh-ai-gateway-infra(89d37cf6-8273-433a-8a54-a94ae3059a6d)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq" podUID="89d37cf6-8273-433a-8a54-a94ae3059a6d" Jul 19 08:18:11.166459 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:18:11.166421 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq" Jul 19 08:18:11.636775 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:18:11.636747 2574 scope.go:117] "RemoveContainer" containerID="8c2e73973e40e2465bb79862e37e8935962b59049a75e7b80e8928a15faedc7c" Jul 19 08:18:11.637171 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:18:11.636930 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq_odh-ai-gateway-infra(89d37cf6-8273-433a-8a54-a94ae3059a6d)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq" podUID="89d37cf6-8273-433a-8a54-a94ae3059a6d" Jul 19 08:18:17.962857 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:18:17.962808 2574 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq" Jul 19 08:18:17.963348 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:18:17.963232 2574 scope.go:117] "RemoveContainer" containerID="8c2e73973e40e2465bb79862e37e8935962b59049a75e7b80e8928a15faedc7c" Jul 19 08:18:17.963442 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:18:17.963422 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq_odh-ai-gateway-infra(89d37cf6-8273-433a-8a54-a94ae3059a6d)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq" podUID="89d37cf6-8273-433a-8a54-a94ae3059a6d" Jul 19 08:18:30.558945 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:18:30.558916 2574 scope.go:117] "RemoveContainer" containerID="8c2e73973e40e2465bb79862e37e8935962b59049a75e7b80e8928a15faedc7c" Jul 19 08:18:30.559367 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:18:30.559129 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq_odh-ai-gateway-infra(89d37cf6-8273-433a-8a54-a94ae3059a6d)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq" podUID="89d37cf6-8273-433a-8a54-a94ae3059a6d" Jul 19 08:18:43.552797 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:18:43.552757 2574 scope.go:117] "RemoveContainer" containerID="8c2e73973e40e2465bb79862e37e8935962b59049a75e7b80e8928a15faedc7c" Jul 19 08:18:43.553387 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:18:43.552967 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq_odh-ai-gateway-infra(89d37cf6-8273-433a-8a54-a94ae3059a6d)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq" podUID="89d37cf6-8273-433a-8a54-a94ae3059a6d" Jul 19 08:18:50.280099 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:18:50.280056 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:18:51.472401 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:18:51.472363 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:18:53.472188 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:18:53.472149 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:18:53.575393 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:18:53.575352 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:18:58.552915 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:18:58.552882 2574 scope.go:117] "RemoveContainer" containerID="8c2e73973e40e2465bb79862e37e8935962b59049a75e7b80e8928a15faedc7c" Jul 19 08:18:58.553327 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:18:58.553078 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq_odh-ai-gateway-infra(89d37cf6-8273-433a-8a54-a94ae3059a6d)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq" podUID="89d37cf6-8273-433a-8a54-a94ae3059a6d" Jul 19 08:19:13.552739 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:19:13.552703 2574 scope.go:117] "RemoveContainer" containerID="8c2e73973e40e2465bb79862e37e8935962b59049a75e7b80e8928a15faedc7c" Jul 19 08:19:13.553160 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:19:13.552898 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq_odh-ai-gateway-infra(89d37cf6-8273-433a-8a54-a94ae3059a6d)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq" podUID="89d37cf6-8273-433a-8a54-a94ae3059a6d" Jul 19 08:19:27.553266 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:19:27.553233 2574 scope.go:117] "RemoveContainer" containerID="8c2e73973e40e2465bb79862e37e8935962b59049a75e7b80e8928a15faedc7c" Jul 19 08:19:27.553742 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:19:27.553488 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq_odh-ai-gateway-infra(89d37cf6-8273-433a-8a54-a94ae3059a6d)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq" podUID="89d37cf6-8273-433a-8a54-a94ae3059a6d" Jul 19 08:19:40.553366 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:19:40.553281 2574 scope.go:117] "RemoveContainer" containerID="8c2e73973e40e2465bb79862e37e8935962b59049a75e7b80e8928a15faedc7c" Jul 19 08:19:40.553802 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:19:40.553494 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq_odh-ai-gateway-infra(89d37cf6-8273-433a-8a54-a94ae3059a6d)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq" podUID="89d37cf6-8273-433a-8a54-a94ae3059a6d" Jul 19 08:19:53.552933 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:19:53.552893 2574 scope.go:117] "RemoveContainer" containerID="8c2e73973e40e2465bb79862e37e8935962b59049a75e7b80e8928a15faedc7c" Jul 19 08:19:53.553368 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:19:53.553151 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq_odh-ai-gateway-infra(89d37cf6-8273-433a-8a54-a94ae3059a6d)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq" podUID="89d37cf6-8273-433a-8a54-a94ae3059a6d" Jul 19 08:20:06.552978 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:20:06.552947 2574 scope.go:117] "RemoveContainer" containerID="8c2e73973e40e2465bb79862e37e8935962b59049a75e7b80e8928a15faedc7c" Jul 19 08:20:06.553593 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:20:06.553143 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq_odh-ai-gateway-infra(89d37cf6-8273-433a-8a54-a94ae3059a6d)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq" podUID="89d37cf6-8273-433a-8a54-a94ae3059a6d" Jul 19 08:20:21.552910 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:20:21.552872 2574 scope.go:117] "RemoveContainer" containerID="8c2e73973e40e2465bb79862e37e8935962b59049a75e7b80e8928a15faedc7c" Jul 19 08:20:21.555311 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:20:21.553075 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq_odh-ai-gateway-infra(89d37cf6-8273-433a-8a54-a94ae3059a6d)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq" podUID="89d37cf6-8273-433a-8a54-a94ae3059a6d" Jul 19 08:20:33.552730 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:20:33.552692 2574 scope.go:117] "RemoveContainer" containerID="8c2e73973e40e2465bb79862e37e8935962b59049a75e7b80e8928a15faedc7c" Jul 19 08:20:33.553132 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:20:33.552873 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq_odh-ai-gateway-infra(89d37cf6-8273-433a-8a54-a94ae3059a6d)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq" podUID="89d37cf6-8273-433a-8a54-a94ae3059a6d" Jul 19 08:20:47.553394 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:20:47.553364 2574 scope.go:117] "RemoveContainer" containerID="8c2e73973e40e2465bb79862e37e8935962b59049a75e7b80e8928a15faedc7c" Jul 19 08:20:47.553869 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:20:47.553590 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq_odh-ai-gateway-infra(89d37cf6-8273-433a-8a54-a94ae3059a6d)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq" podUID="89d37cf6-8273-433a-8a54-a94ae3059a6d" Jul 19 08:20:55.291034 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:20:55.290986 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:21:01.553462 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:01.553431 2574 scope.go:117] "RemoveContainer" containerID="8c2e73973e40e2465bb79862e37e8935962b59049a75e7b80e8928a15faedc7c" Jul 19 08:21:01.553925 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:01.553793 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 19 08:21:02.482228 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:02.482190 2574 generic.go:356] "Generic (PLEG): container finished" podID="89d37cf6-8273-433a-8a54-a94ae3059a6d" containerID="503c22096e55b099de432142585a4b215c585f1afab5d2701e85fbfb63d6bdce" exitCode=1 Jul 19 08:21:02.482426 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:02.482259 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq" event={"ID":"89d37cf6-8273-433a-8a54-a94ae3059a6d","Type":"ContainerDied","Data":"503c22096e55b099de432142585a4b215c585f1afab5d2701e85fbfb63d6bdce"} Jul 19 08:21:02.482426 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:02.482319 2574 scope.go:117] "RemoveContainer" containerID="8c2e73973e40e2465bb79862e37e8935962b59049a75e7b80e8928a15faedc7c" Jul 19 08:21:02.482814 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:02.482715 2574 scope.go:117] "RemoveContainer" containerID="503c22096e55b099de432142585a4b215c585f1afab5d2701e85fbfb63d6bdce" Jul 19 08:21:02.482956 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:21:02.482932 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq_odh-ai-gateway-infra(89d37cf6-8273-433a-8a54-a94ae3059a6d)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq" podUID="89d37cf6-8273-433a-8a54-a94ae3059a6d" Jul 19 08:21:02.876893 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:02.876852 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-2a60bf57-openshift-default-5bdb955d89-sclw2"] Jul 19 08:21:02.877372 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:02.877107 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-2a60bf57-openshift-default-5bdb955d89-sclw2" podUID="a81a2508-608d-43ea-89fa-905c6539f8f1" containerName="istio-proxy" containerID="cri-o://b1ffbbec0d8a0932d7652a9fd632a57773a1b286cd43b6dee085f301eef917e0" gracePeriod=30 Jul 19 08:21:02.892833 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:02.892785 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:21:02.999997 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:02.999951 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:21:05.972636 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:05.972533 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 08:21:06.166168 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:06.166133 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq" Jul 19 08:21:06.166606 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:06.166589 2574 scope.go:117] "RemoveContainer" containerID="503c22096e55b099de432142585a4b215c585f1afab5d2701e85fbfb63d6bdce" Jul 19 08:21:06.166780 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:21:06.166761 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq_odh-ai-gateway-infra(89d37cf6-8273-433a-8a54-a94ae3059a6d)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq" podUID="89d37cf6-8273-433a-8a54-a94ae3059a6d" Jul 19 08:21:07.963204 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:07.963155 2574 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq" Jul 19 08:21:07.963623 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:07.963605 2574 scope.go:117] "RemoveContainer" containerID="503c22096e55b099de432142585a4b215c585f1afab5d2701e85fbfb63d6bdce" Jul 19 08:21:07.963819 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:21:07.963796 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq_odh-ai-gateway-infra(89d37cf6-8273-433a-8a54-a94ae3059a6d)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq" podUID="89d37cf6-8273-433a-8a54-a94ae3059a6d" Jul 19 08:21:08.140669 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:08.140643 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2a60bf57-openshift-default-5bdb955d89-sclw2" Jul 19 08:21:08.223858 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:08.223825 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a81a2508-608d-43ea-89fa-905c6539f8f1-istio-token\") pod \"a81a2508-608d-43ea-89fa-905c6539f8f1\" (UID: \"a81a2508-608d-43ea-89fa-905c6539f8f1\") " Jul 19 08:21:08.224057 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:08.223893 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a81a2508-608d-43ea-89fa-905c6539f8f1-istio-podinfo\") pod \"a81a2508-608d-43ea-89fa-905c6539f8f1\" (UID: \"a81a2508-608d-43ea-89fa-905c6539f8f1\") " Jul 19 08:21:08.224057 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:08.223926 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a81a2508-608d-43ea-89fa-905c6539f8f1-workload-certs\") pod \"a81a2508-608d-43ea-89fa-905c6539f8f1\" (UID: \"a81a2508-608d-43ea-89fa-905c6539f8f1\") " Jul 19 08:21:08.224057 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:08.223970 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a81a2508-608d-43ea-89fa-905c6539f8f1-istiod-ca-cert\") pod \"a81a2508-608d-43ea-89fa-905c6539f8f1\" (UID: \"a81a2508-608d-43ea-89fa-905c6539f8f1\") " Jul 19 08:21:08.224057 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:08.224028 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a81a2508-608d-43ea-89fa-905c6539f8f1-istio-data\") pod \"a81a2508-608d-43ea-89fa-905c6539f8f1\" (UID: \"a81a2508-608d-43ea-89fa-905c6539f8f1\") " Jul 19 08:21:08.224057 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:08.224057 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a81a2508-608d-43ea-89fa-905c6539f8f1-workload-socket\") pod \"a81a2508-608d-43ea-89fa-905c6539f8f1\" (UID: \"a81a2508-608d-43ea-89fa-905c6539f8f1\") " Jul 19 08:21:08.224337 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:08.224099 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fxsvm\" (UniqueName: \"kubernetes.io/projected/a81a2508-608d-43ea-89fa-905c6539f8f1-kube-api-access-fxsvm\") pod \"a81a2508-608d-43ea-89fa-905c6539f8f1\" (UID: \"a81a2508-608d-43ea-89fa-905c6539f8f1\") " Jul 19 08:21:08.224337 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:08.224140 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a81a2508-608d-43ea-89fa-905c6539f8f1-credential-socket\") pod \"a81a2508-608d-43ea-89fa-905c6539f8f1\" (UID: \"a81a2508-608d-43ea-89fa-905c6539f8f1\") " Jul 19 08:21:08.224337 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:08.224167 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a81a2508-608d-43ea-89fa-905c6539f8f1-istio-envoy\") pod \"a81a2508-608d-43ea-89fa-905c6539f8f1\" (UID: \"a81a2508-608d-43ea-89fa-905c6539f8f1\") " Jul 19 08:21:08.224337 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:08.224263 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a81a2508-608d-43ea-89fa-905c6539f8f1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a81a2508-608d-43ea-89fa-905c6539f8f1" (UID: "a81a2508-608d-43ea-89fa-905c6539f8f1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 08:21:08.224571 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:08.224438 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a81a2508-608d-43ea-89fa-905c6539f8f1-workload-certs\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:21:08.224749 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:08.224671 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a81a2508-608d-43ea-89fa-905c6539f8f1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a81a2508-608d-43ea-89fa-905c6539f8f1" (UID: "a81a2508-608d-43ea-89fa-905c6539f8f1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 08:21:08.225419 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:08.225382 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a81a2508-608d-43ea-89fa-905c6539f8f1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a81a2508-608d-43ea-89fa-905c6539f8f1" (UID: "a81a2508-608d-43ea-89fa-905c6539f8f1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 08:21:08.225419 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:08.225393 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a81a2508-608d-43ea-89fa-905c6539f8f1-istio-data" (OuterVolumeSpecName: "istio-data") pod "a81a2508-608d-43ea-89fa-905c6539f8f1" (UID: "a81a2508-608d-43ea-89fa-905c6539f8f1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 08:21:08.225604 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:08.225467 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a81a2508-608d-43ea-89fa-905c6539f8f1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a81a2508-608d-43ea-89fa-905c6539f8f1" (UID: "a81a2508-608d-43ea-89fa-905c6539f8f1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 08:21:08.227119 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:08.227080 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a81a2508-608d-43ea-89fa-905c6539f8f1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a81a2508-608d-43ea-89fa-905c6539f8f1" (UID: "a81a2508-608d-43ea-89fa-905c6539f8f1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 19 08:21:08.227376 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:08.227346 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a81a2508-608d-43ea-89fa-905c6539f8f1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a81a2508-608d-43ea-89fa-905c6539f8f1" (UID: "a81a2508-608d-43ea-89fa-905c6539f8f1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 08:21:08.227762 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:08.227733 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a81a2508-608d-43ea-89fa-905c6539f8f1-istio-token" (OuterVolumeSpecName: "istio-token") pod "a81a2508-608d-43ea-89fa-905c6539f8f1" (UID: "a81a2508-608d-43ea-89fa-905c6539f8f1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 08:21:08.228091 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:08.228068 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a81a2508-608d-43ea-89fa-905c6539f8f1-kube-api-access-fxsvm" (OuterVolumeSpecName: "kube-api-access-fxsvm") pod "a81a2508-608d-43ea-89fa-905c6539f8f1" (UID: "a81a2508-608d-43ea-89fa-905c6539f8f1"). InnerVolumeSpecName "kube-api-access-fxsvm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 08:21:08.326064 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:08.326013 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a81a2508-608d-43ea-89fa-905c6539f8f1-istio-data\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:21:08.326064 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:08.326053 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a81a2508-608d-43ea-89fa-905c6539f8f1-workload-socket\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:21:08.326064 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:08.326067 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxsvm\" (UniqueName: \"kubernetes.io/projected/a81a2508-608d-43ea-89fa-905c6539f8f1-kube-api-access-fxsvm\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:21:08.326371 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:08.326080 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a81a2508-608d-43ea-89fa-905c6539f8f1-credential-socket\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:21:08.326371 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:08.326096 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a81a2508-608d-43ea-89fa-905c6539f8f1-istio-envoy\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:21:08.326371 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:08.326108 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a81a2508-608d-43ea-89fa-905c6539f8f1-istio-token\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:21:08.326371 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:08.326121 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a81a2508-608d-43ea-89fa-905c6539f8f1-istio-podinfo\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:21:08.326371 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:08.326134 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a81a2508-608d-43ea-89fa-905c6539f8f1-istiod-ca-cert\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:21:08.393716 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:08.393677 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq"] Jul 19 08:21:08.413574 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:08.413506 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:21:08.514905 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:08.514873 2574 generic.go:356] "Generic (PLEG): container finished" podID="a81a2508-608d-43ea-89fa-905c6539f8f1" containerID="b1ffbbec0d8a0932d7652a9fd632a57773a1b286cd43b6dee085f301eef917e0" exitCode=0 Jul 19 08:21:08.515102 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:08.514920 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2a60bf57-openshift-default-5bdb955d89-sclw2" Jul 19 08:21:08.515102 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:08.514957 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2a60bf57-openshift-default-5bdb955d89-sclw2" event={"ID":"a81a2508-608d-43ea-89fa-905c6539f8f1","Type":"ContainerDied","Data":"b1ffbbec0d8a0932d7652a9fd632a57773a1b286cd43b6dee085f301eef917e0"} Jul 19 08:21:08.515102 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:08.515016 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2a60bf57-openshift-default-5bdb955d89-sclw2" event={"ID":"a81a2508-608d-43ea-89fa-905c6539f8f1","Type":"ContainerDied","Data":"354ec0ae7a371184cc41cf50ebc5572d1194a6096e156d8043167daa0563dafb"} Jul 19 08:21:08.515102 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:08.515048 2574 scope.go:117] "RemoveContainer" containerID="b1ffbbec0d8a0932d7652a9fd632a57773a1b286cd43b6dee085f301eef917e0" Jul 19 08:21:08.526232 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:08.526212 2574 scope.go:117] "RemoveContainer" containerID="16b27542354d40a618de6705149c4c03ae7220f6f6cd59254b36f89bd830af78" Jul 19 08:21:08.536897 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:08.536876 2574 scope.go:117] "RemoveContainer" containerID="b1ffbbec0d8a0932d7652a9fd632a57773a1b286cd43b6dee085f301eef917e0" Jul 19 08:21:08.537260 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:21:08.537239 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1ffbbec0d8a0932d7652a9fd632a57773a1b286cd43b6dee085f301eef917e0\": container with ID starting with b1ffbbec0d8a0932d7652a9fd632a57773a1b286cd43b6dee085f301eef917e0 not found: ID does not exist" containerID="b1ffbbec0d8a0932d7652a9fd632a57773a1b286cd43b6dee085f301eef917e0" Jul 19 08:21:08.537359 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:08.537274 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1ffbbec0d8a0932d7652a9fd632a57773a1b286cd43b6dee085f301eef917e0"} err="failed to get container status \"b1ffbbec0d8a0932d7652a9fd632a57773a1b286cd43b6dee085f301eef917e0\": rpc error: code = NotFound desc = could not find container \"b1ffbbec0d8a0932d7652a9fd632a57773a1b286cd43b6dee085f301eef917e0\": container with ID starting with b1ffbbec0d8a0932d7652a9fd632a57773a1b286cd43b6dee085f301eef917e0 not found: ID does not exist" Jul 19 08:21:08.537359 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:08.537299 2574 scope.go:117] "RemoveContainer" containerID="16b27542354d40a618de6705149c4c03ae7220f6f6cd59254b36f89bd830af78" Jul 19 08:21:08.537598 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:21:08.537578 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16b27542354d40a618de6705149c4c03ae7220f6f6cd59254b36f89bd830af78\": container with ID starting with 16b27542354d40a618de6705149c4c03ae7220f6f6cd59254b36f89bd830af78 not found: ID does not exist" containerID="16b27542354d40a618de6705149c4c03ae7220f6f6cd59254b36f89bd830af78" Jul 19 08:21:08.537682 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:08.537606 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16b27542354d40a618de6705149c4c03ae7220f6f6cd59254b36f89bd830af78"} err="failed to get container status \"16b27542354d40a618de6705149c4c03ae7220f6f6cd59254b36f89bd830af78\": rpc error: code = NotFound desc = could not find container \"16b27542354d40a618de6705149c4c03ae7220f6f6cd59254b36f89bd830af78\": container with ID starting with 16b27542354d40a618de6705149c4c03ae7220f6f6cd59254b36f89bd830af78 not found: ID does not exist" Jul 19 08:21:08.545885 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:08.545863 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-2a60bf57-openshift-default-5bdb955d89-sclw2"] Jul 19 08:21:08.549705 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:08.549677 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-2a60bf57-openshift-default-5bdb955d89-sclw2"] Jul 19 08:21:08.558726 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:08.558702 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a81a2508-608d-43ea-89fa-905c6539f8f1" path="/var/lib/kubelet/pods/a81a2508-608d-43ea-89fa-905c6539f8f1/volumes" Jul 19 08:21:08.649430 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:08.649404 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq" Jul 19 08:21:08.662843 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:08.662814 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:21:08.728835 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:08.728797 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89d37cf6-8273-433a-8a54-a94ae3059a6d-maas-api-tls\") pod \"89d37cf6-8273-433a-8a54-a94ae3059a6d\" (UID: \"89d37cf6-8273-433a-8a54-a94ae3059a6d\") " Jul 19 08:21:08.728835 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:08.728842 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89d37cf6-8273-433a-8a54-a94ae3059a6d-openshift-service-ca\") pod \"89d37cf6-8273-433a-8a54-a94ae3059a6d\" (UID: \"89d37cf6-8273-433a-8a54-a94ae3059a6d\") " Jul 19 08:21:08.729119 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:08.728922 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lmm9t\" (UniqueName: \"kubernetes.io/projected/89d37cf6-8273-433a-8a54-a94ae3059a6d-kube-api-access-lmm9t\") pod \"89d37cf6-8273-433a-8a54-a94ae3059a6d\" (UID: \"89d37cf6-8273-433a-8a54-a94ae3059a6d\") " Jul 19 08:21:08.729202 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:08.729175 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89d37cf6-8273-433a-8a54-a94ae3059a6d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "89d37cf6-8273-433a-8a54-a94ae3059a6d" (UID: "89d37cf6-8273-433a-8a54-a94ae3059a6d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 08:21:08.730989 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:08.730967 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89d37cf6-8273-433a-8a54-a94ae3059a6d-kube-api-access-lmm9t" (OuterVolumeSpecName: "kube-api-access-lmm9t") pod "89d37cf6-8273-433a-8a54-a94ae3059a6d" (UID: "89d37cf6-8273-433a-8a54-a94ae3059a6d"). InnerVolumeSpecName "kube-api-access-lmm9t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 08:21:08.731076 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:08.731044 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89d37cf6-8273-433a-8a54-a94ae3059a6d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "89d37cf6-8273-433a-8a54-a94ae3059a6d" (UID: "89d37cf6-8273-433a-8a54-a94ae3059a6d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 08:21:08.830115 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:08.830027 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89d37cf6-8273-433a-8a54-a94ae3059a6d-maas-api-tls\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:21:08.830115 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:08.830060 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89d37cf6-8273-433a-8a54-a94ae3059a6d-openshift-service-ca\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:21:08.830115 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:08.830071 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmm9t\" (UniqueName: \"kubernetes.io/projected/89d37cf6-8273-433a-8a54-a94ae3059a6d-kube-api-access-lmm9t\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:21:09.522422 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:09.522384 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq" Jul 19 08:21:09.522422 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:09.522413 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq" event={"ID":"89d37cf6-8273-433a-8a54-a94ae3059a6d","Type":"ContainerDied","Data":"501266eb5dac8680d3fdb16d26ae805ca02593a2d5f1b9d40a89f9f1a0a95ce7"} Jul 19 08:21:09.523018 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:09.522480 2574 scope.go:117] "RemoveContainer" containerID="503c22096e55b099de432142585a4b215c585f1afab5d2701e85fbfb63d6bdce" Jul 19 08:21:09.544283 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:09.544255 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq"] Jul 19 08:21:09.548301 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:09.548275 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2a60bf57-5f9d5f9896-hdxfq"] Jul 19 08:21:10.559867 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:10.559826 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89d37cf6-8273-433a-8a54-a94ae3059a6d" path="/var/lib/kubelet/pods/89d37cf6-8273-433a-8a54-a94ae3059a6d/volumes" Jul 19 08:21:14.058800 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:14.058761 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:21:14.158686 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:14.158644 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:21:14.256676 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:14.256646 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-fdbad2e2-openshift-default-5cf7ff57b7-g8sqd"] Jul 19 08:21:14.257331 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:14.257301 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89d37cf6-8273-433a-8a54-a94ae3059a6d" containerName="maas-api" Jul 19 08:21:14.257331 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:14.257328 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="89d37cf6-8273-433a-8a54-a94ae3059a6d" containerName="maas-api" Jul 19 08:21:14.257509 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:14.257341 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d21b86e6-517d-44f3-9422-5ba22f78afe8" containerName="revoke-keys" Jul 19 08:21:14.257509 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:14.257347 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="d21b86e6-517d-44f3-9422-5ba22f78afe8" containerName="revoke-keys" Jul 19 08:21:14.257509 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:14.257354 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d21b86e6-517d-44f3-9422-5ba22f78afe8" containerName="revoke-keys" Jul 19 08:21:14.257509 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:14.257359 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="d21b86e6-517d-44f3-9422-5ba22f78afe8" containerName="revoke-keys" Jul 19 08:21:14.257509 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:14.257366 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89d37cf6-8273-433a-8a54-a94ae3059a6d" containerName="maas-api" Jul 19 08:21:14.257509 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:14.257371 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="89d37cf6-8273-433a-8a54-a94ae3059a6d" containerName="maas-api" Jul 19 08:21:14.257509 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:14.257380 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89d37cf6-8273-433a-8a54-a94ae3059a6d" containerName="maas-api" Jul 19 08:21:14.257509 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:14.257389 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="89d37cf6-8273-433a-8a54-a94ae3059a6d" containerName="maas-api" Jul 19 08:21:14.257509 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:14.257399 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d21b86e6-517d-44f3-9422-5ba22f78afe8" containerName="revoke-keys" Jul 19 08:21:14.257509 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:14.257404 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="d21b86e6-517d-44f3-9422-5ba22f78afe8" containerName="revoke-keys" Jul 19 08:21:14.257509 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:14.257414 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a81a2508-608d-43ea-89fa-905c6539f8f1" containerName="istio-proxy" Jul 19 08:21:14.257509 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:14.257419 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="a81a2508-608d-43ea-89fa-905c6539f8f1" containerName="istio-proxy" Jul 19 08:21:14.257509 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:14.257436 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89d37cf6-8273-433a-8a54-a94ae3059a6d" containerName="maas-api" Jul 19 08:21:14.257509 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:14.257441 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="89d37cf6-8273-433a-8a54-a94ae3059a6d" containerName="maas-api" Jul 19 08:21:14.257509 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:14.257512 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="89d37cf6-8273-433a-8a54-a94ae3059a6d" containerName="maas-api" Jul 19 08:21:14.258093 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:14.257537 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="89d37cf6-8273-433a-8a54-a94ae3059a6d" containerName="maas-api" Jul 19 08:21:14.258093 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:14.257547 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="89d37cf6-8273-433a-8a54-a94ae3059a6d" containerName="maas-api" Jul 19 08:21:14.258093 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:14.257553 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="89d37cf6-8273-433a-8a54-a94ae3059a6d" containerName="maas-api" Jul 19 08:21:14.258093 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:14.257558 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="a81a2508-608d-43ea-89fa-905c6539f8f1" containerName="istio-proxy" Jul 19 08:21:14.258093 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:14.257564 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="d21b86e6-517d-44f3-9422-5ba22f78afe8" containerName="revoke-keys" Jul 19 08:21:14.258093 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:14.257569 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="a81a2508-608d-43ea-89fa-905c6539f8f1" containerName="istio-proxy" Jul 19 08:21:14.258093 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:14.257574 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="89d37cf6-8273-433a-8a54-a94ae3059a6d" containerName="maas-api" Jul 19 08:21:14.258093 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:14.257581 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="d21b86e6-517d-44f3-9422-5ba22f78afe8" containerName="revoke-keys" Jul 19 08:21:14.262166 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:14.262146 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-fdbad2e2-openshift-default-5cf7ff57b7-g8sqd" Jul 19 08:21:14.264696 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:14.264664 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-fdbad2e2-openshift-default-dockercfg-28rzn\"" Jul 19 08:21:14.264818 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:14.264766 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 19 08:21:14.272447 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:14.272418 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-fdbad2e2-openshift-default-5cf7ff57b7-g8sqd"] Jul 19 08:21:14.294347 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:14.294313 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:21:14.380157 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:14.380076 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dac3f27a-2ab3-4707-84b9-ed590c895b2c-istio-envoy\") pod \"e2e-derive-fdbad2e2-openshift-default-5cf7ff57b7-g8sqd\" (UID: \"dac3f27a-2ab3-4707-84b9-ed590c895b2c\") " pod="openshift-ingress/e2e-derive-fdbad2e2-openshift-default-5cf7ff57b7-g8sqd" Jul 19 08:21:14.380157 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:14.380113 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dac3f27a-2ab3-4707-84b9-ed590c895b2c-istio-podinfo\") pod \"e2e-derive-fdbad2e2-openshift-default-5cf7ff57b7-g8sqd\" (UID: \"dac3f27a-2ab3-4707-84b9-ed590c895b2c\") " pod="openshift-ingress/e2e-derive-fdbad2e2-openshift-default-5cf7ff57b7-g8sqd" Jul 19 08:21:14.380354 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:14.380191 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dac3f27a-2ab3-4707-84b9-ed590c895b2c-workload-socket\") pod \"e2e-derive-fdbad2e2-openshift-default-5cf7ff57b7-g8sqd\" (UID: \"dac3f27a-2ab3-4707-84b9-ed590c895b2c\") " pod="openshift-ingress/e2e-derive-fdbad2e2-openshift-default-5cf7ff57b7-g8sqd" Jul 19 08:21:14.380354 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:14.380230 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dac3f27a-2ab3-4707-84b9-ed590c895b2c-istio-token\") pod \"e2e-derive-fdbad2e2-openshift-default-5cf7ff57b7-g8sqd\" (UID: \"dac3f27a-2ab3-4707-84b9-ed590c895b2c\") " pod="openshift-ingress/e2e-derive-fdbad2e2-openshift-default-5cf7ff57b7-g8sqd" Jul 19 08:21:14.380354 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:14.380255 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dac3f27a-2ab3-4707-84b9-ed590c895b2c-workload-certs\") pod \"e2e-derive-fdbad2e2-openshift-default-5cf7ff57b7-g8sqd\" (UID: \"dac3f27a-2ab3-4707-84b9-ed590c895b2c\") " pod="openshift-ingress/e2e-derive-fdbad2e2-openshift-default-5cf7ff57b7-g8sqd" Jul 19 08:21:14.380467 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:14.380365 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rw9m\" (UniqueName: \"kubernetes.io/projected/dac3f27a-2ab3-4707-84b9-ed590c895b2c-kube-api-access-9rw9m\") pod \"e2e-derive-fdbad2e2-openshift-default-5cf7ff57b7-g8sqd\" (UID: \"dac3f27a-2ab3-4707-84b9-ed590c895b2c\") " pod="openshift-ingress/e2e-derive-fdbad2e2-openshift-default-5cf7ff57b7-g8sqd" Jul 19 08:21:14.380467 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:14.380396 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dac3f27a-2ab3-4707-84b9-ed590c895b2c-istio-data\") pod \"e2e-derive-fdbad2e2-openshift-default-5cf7ff57b7-g8sqd\" (UID: \"dac3f27a-2ab3-4707-84b9-ed590c895b2c\") " pod="openshift-ingress/e2e-derive-fdbad2e2-openshift-default-5cf7ff57b7-g8sqd" Jul 19 08:21:14.380467 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:14.380427 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dac3f27a-2ab3-4707-84b9-ed590c895b2c-istiod-ca-cert\") pod \"e2e-derive-fdbad2e2-openshift-default-5cf7ff57b7-g8sqd\" (UID: \"dac3f27a-2ab3-4707-84b9-ed590c895b2c\") " pod="openshift-ingress/e2e-derive-fdbad2e2-openshift-default-5cf7ff57b7-g8sqd" Jul 19 08:21:14.380467 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:14.380457 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dac3f27a-2ab3-4707-84b9-ed590c895b2c-credential-socket\") pod \"e2e-derive-fdbad2e2-openshift-default-5cf7ff57b7-g8sqd\" (UID: \"dac3f27a-2ab3-4707-84b9-ed590c895b2c\") " pod="openshift-ingress/e2e-derive-fdbad2e2-openshift-default-5cf7ff57b7-g8sqd" Jul 19 08:21:14.480839 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:14.480798 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dac3f27a-2ab3-4707-84b9-ed590c895b2c-istio-data\") pod \"e2e-derive-fdbad2e2-openshift-default-5cf7ff57b7-g8sqd\" (UID: \"dac3f27a-2ab3-4707-84b9-ed590c895b2c\") " pod="openshift-ingress/e2e-derive-fdbad2e2-openshift-default-5cf7ff57b7-g8sqd" Jul 19 08:21:14.480839 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:14.480840 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dac3f27a-2ab3-4707-84b9-ed590c895b2c-istiod-ca-cert\") pod \"e2e-derive-fdbad2e2-openshift-default-5cf7ff57b7-g8sqd\" (UID: \"dac3f27a-2ab3-4707-84b9-ed590c895b2c\") " pod="openshift-ingress/e2e-derive-fdbad2e2-openshift-default-5cf7ff57b7-g8sqd" Jul 19 08:21:14.481084 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:14.480865 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dac3f27a-2ab3-4707-84b9-ed590c895b2c-credential-socket\") pod \"e2e-derive-fdbad2e2-openshift-default-5cf7ff57b7-g8sqd\" (UID: \"dac3f27a-2ab3-4707-84b9-ed590c895b2c\") " pod="openshift-ingress/e2e-derive-fdbad2e2-openshift-default-5cf7ff57b7-g8sqd" Jul 19 08:21:14.481084 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:14.480915 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dac3f27a-2ab3-4707-84b9-ed590c895b2c-istio-envoy\") pod \"e2e-derive-fdbad2e2-openshift-default-5cf7ff57b7-g8sqd\" (UID: \"dac3f27a-2ab3-4707-84b9-ed590c895b2c\") " pod="openshift-ingress/e2e-derive-fdbad2e2-openshift-default-5cf7ff57b7-g8sqd" Jul 19 08:21:14.481084 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:14.480939 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dac3f27a-2ab3-4707-84b9-ed590c895b2c-istio-podinfo\") pod \"e2e-derive-fdbad2e2-openshift-default-5cf7ff57b7-g8sqd\" (UID: \"dac3f27a-2ab3-4707-84b9-ed590c895b2c\") " pod="openshift-ingress/e2e-derive-fdbad2e2-openshift-default-5cf7ff57b7-g8sqd" Jul 19 08:21:14.481084 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:14.480978 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dac3f27a-2ab3-4707-84b9-ed590c895b2c-workload-socket\") pod \"e2e-derive-fdbad2e2-openshift-default-5cf7ff57b7-g8sqd\" (UID: \"dac3f27a-2ab3-4707-84b9-ed590c895b2c\") " pod="openshift-ingress/e2e-derive-fdbad2e2-openshift-default-5cf7ff57b7-g8sqd" Jul 19 08:21:14.481084 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:14.481005 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dac3f27a-2ab3-4707-84b9-ed590c895b2c-istio-token\") pod \"e2e-derive-fdbad2e2-openshift-default-5cf7ff57b7-g8sqd\" (UID: \"dac3f27a-2ab3-4707-84b9-ed590c895b2c\") " pod="openshift-ingress/e2e-derive-fdbad2e2-openshift-default-5cf7ff57b7-g8sqd" Jul 19 08:21:14.481084 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:14.481032 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dac3f27a-2ab3-4707-84b9-ed590c895b2c-workload-certs\") pod \"e2e-derive-fdbad2e2-openshift-default-5cf7ff57b7-g8sqd\" (UID: \"dac3f27a-2ab3-4707-84b9-ed590c895b2c\") " pod="openshift-ingress/e2e-derive-fdbad2e2-openshift-default-5cf7ff57b7-g8sqd" Jul 19 08:21:14.481410 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:14.481105 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rw9m\" (UniqueName: \"kubernetes.io/projected/dac3f27a-2ab3-4707-84b9-ed590c895b2c-kube-api-access-9rw9m\") pod \"e2e-derive-fdbad2e2-openshift-default-5cf7ff57b7-g8sqd\" (UID: \"dac3f27a-2ab3-4707-84b9-ed590c895b2c\") " pod="openshift-ingress/e2e-derive-fdbad2e2-openshift-default-5cf7ff57b7-g8sqd" Jul 19 08:21:14.481410 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:14.481255 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dac3f27a-2ab3-4707-84b9-ed590c895b2c-istio-data\") pod \"e2e-derive-fdbad2e2-openshift-default-5cf7ff57b7-g8sqd\" (UID: \"dac3f27a-2ab3-4707-84b9-ed590c895b2c\") " pod="openshift-ingress/e2e-derive-fdbad2e2-openshift-default-5cf7ff57b7-g8sqd" Jul 19 08:21:14.481410 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:14.481284 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dac3f27a-2ab3-4707-84b9-ed590c895b2c-credential-socket\") pod \"e2e-derive-fdbad2e2-openshift-default-5cf7ff57b7-g8sqd\" (UID: \"dac3f27a-2ab3-4707-84b9-ed590c895b2c\") " pod="openshift-ingress/e2e-derive-fdbad2e2-openshift-default-5cf7ff57b7-g8sqd" Jul 19 08:21:14.481410 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:14.481359 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dac3f27a-2ab3-4707-84b9-ed590c895b2c-workload-socket\") pod \"e2e-derive-fdbad2e2-openshift-default-5cf7ff57b7-g8sqd\" (UID: \"dac3f27a-2ab3-4707-84b9-ed590c895b2c\") " pod="openshift-ingress/e2e-derive-fdbad2e2-openshift-default-5cf7ff57b7-g8sqd" Jul 19 08:21:14.481572 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:14.481509 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dac3f27a-2ab3-4707-84b9-ed590c895b2c-workload-certs\") pod \"e2e-derive-fdbad2e2-openshift-default-5cf7ff57b7-g8sqd\" (UID: \"dac3f27a-2ab3-4707-84b9-ed590c895b2c\") " pod="openshift-ingress/e2e-derive-fdbad2e2-openshift-default-5cf7ff57b7-g8sqd" Jul 19 08:21:14.483213 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:14.483193 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dac3f27a-2ab3-4707-84b9-ed590c895b2c-istio-envoy\") pod \"e2e-derive-fdbad2e2-openshift-default-5cf7ff57b7-g8sqd\" (UID: \"dac3f27a-2ab3-4707-84b9-ed590c895b2c\") " pod="openshift-ingress/e2e-derive-fdbad2e2-openshift-default-5cf7ff57b7-g8sqd" Jul 19 08:21:14.483337 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:14.483319 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dac3f27a-2ab3-4707-84b9-ed590c895b2c-istio-podinfo\") pod \"e2e-derive-fdbad2e2-openshift-default-5cf7ff57b7-g8sqd\" (UID: \"dac3f27a-2ab3-4707-84b9-ed590c895b2c\") " pod="openshift-ingress/e2e-derive-fdbad2e2-openshift-default-5cf7ff57b7-g8sqd" Jul 19 08:21:14.483546 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:14.483506 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 19 08:21:14.488542 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:14.488492 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dac3f27a-2ab3-4707-84b9-ed590c895b2c-istio-token\") pod \"e2e-derive-fdbad2e2-openshift-default-5cf7ff57b7-g8sqd\" (UID: \"dac3f27a-2ab3-4707-84b9-ed590c895b2c\") " pod="openshift-ingress/e2e-derive-fdbad2e2-openshift-default-5cf7ff57b7-g8sqd" Jul 19 08:21:14.488775 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:14.488758 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rw9m\" (UniqueName: \"kubernetes.io/projected/dac3f27a-2ab3-4707-84b9-ed590c895b2c-kube-api-access-9rw9m\") pod \"e2e-derive-fdbad2e2-openshift-default-5cf7ff57b7-g8sqd\" (UID: \"dac3f27a-2ab3-4707-84b9-ed590c895b2c\") " pod="openshift-ingress/e2e-derive-fdbad2e2-openshift-default-5cf7ff57b7-g8sqd" Jul 19 08:21:14.491930 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:14.491911 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dac3f27a-2ab3-4707-84b9-ed590c895b2c-istiod-ca-cert\") pod \"e2e-derive-fdbad2e2-openshift-default-5cf7ff57b7-g8sqd\" (UID: \"dac3f27a-2ab3-4707-84b9-ed590c895b2c\") " pod="openshift-ingress/e2e-derive-fdbad2e2-openshift-default-5cf7ff57b7-g8sqd" Jul 19 08:21:14.579826 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:14.579797 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-fdbad2e2-openshift-default-dockercfg-28rzn\"" Jul 19 08:21:14.587764 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:14.587742 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-fdbad2e2-openshift-default-5cf7ff57b7-g8sqd" Jul 19 08:21:14.736752 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:14.736726 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-fdbad2e2-openshift-default-5cf7ff57b7-g8sqd"] Jul 19 08:21:14.739568 ip-10-0-136-70 kubenswrapper[2574]: W0719 08:21:14.739508 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddac3f27a_2ab3_4707_84b9_ed590c895b2c.slice/crio-bf797a3d58e84513c4d0ca2405313972ff5431512d5e7c23c00f81989dd44513 WatchSource:0}: Error finding container bf797a3d58e84513c4d0ca2405313972ff5431512d5e7c23c00f81989dd44513: Status 404 returned error can't find the container with id bf797a3d58e84513c4d0ca2405313972ff5431512d5e7c23c00f81989dd44513 Jul 19 08:21:14.742112 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:14.742063 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 19 08:21:14.742228 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:14.742171 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 19 08:21:14.742228 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:14.742217 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 19 08:21:15.406022 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:15.405987 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:21:15.506174 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:15.506142 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:21:15.556484 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:15.556447 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-fdbad2e2-openshift-default-5cf7ff57b7-g8sqd" event={"ID":"dac3f27a-2ab3-4707-84b9-ed590c895b2c","Type":"ContainerStarted","Data":"50b6323edf7d51394e4972c91bf1408dd9cb1ed8755c0a18475bf7f91fe6ed14"} Jul 19 08:21:15.556745 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:15.556514 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-fdbad2e2-openshift-default-5cf7ff57b7-g8sqd" event={"ID":"dac3f27a-2ab3-4707-84b9-ed590c895b2c","Type":"ContainerStarted","Data":"bf797a3d58e84513c4d0ca2405313972ff5431512d5e7c23c00f81989dd44513"} Jul 19 08:21:15.579848 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:15.579797 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-fdbad2e2-openshift-default-5cf7ff57b7-g8sqd" podStartSLOduration=1.579779561 podStartE2EDuration="1.579779561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 08:21:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:21:15.578905729 +0000 UTC m=+2341.933215346" watchObservedRunningTime="2026-07-19 08:21:15.579779561 +0000 UTC m=+2341.934089162" Jul 19 08:21:15.588306 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:15.588278 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-fdbad2e2-openshift-default-5cf7ff57b7-g8sqd" Jul 19 08:21:15.593637 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:15.593615 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-fdbad2e2-openshift-default-5cf7ff57b7-g8sqd" Jul 19 08:21:15.652816 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:15.652783 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-fdbad2e2-795d7b46bb-24d7l"] Jul 19 08:21:15.653377 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:15.653364 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89d37cf6-8273-433a-8a54-a94ae3059a6d" containerName="maas-api" Jul 19 08:21:15.653422 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:15.653383 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="89d37cf6-8273-433a-8a54-a94ae3059a6d" containerName="maas-api" Jul 19 08:21:15.653422 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:15.653401 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89d37cf6-8273-433a-8a54-a94ae3059a6d" containerName="maas-api" Jul 19 08:21:15.653422 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:15.653411 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="89d37cf6-8273-433a-8a54-a94ae3059a6d" containerName="maas-api" Jul 19 08:21:15.653536 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:15.653422 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89d37cf6-8273-433a-8a54-a94ae3059a6d" containerName="maas-api" Jul 19 08:21:15.653536 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:15.653432 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="89d37cf6-8273-433a-8a54-a94ae3059a6d" containerName="maas-api" Jul 19 08:21:15.653536 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:15.653453 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a81a2508-608d-43ea-89fa-905c6539f8f1" containerName="istio-proxy" Jul 19 08:21:15.653536 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:15.653462 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="a81a2508-608d-43ea-89fa-905c6539f8f1" containerName="istio-proxy" Jul 19 08:21:15.653656 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:15.653574 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="89d37cf6-8273-433a-8a54-a94ae3059a6d" containerName="maas-api" Jul 19 08:21:15.653656 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:15.653587 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="89d37cf6-8273-433a-8a54-a94ae3059a6d" containerName="maas-api" Jul 19 08:21:15.653656 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:15.653600 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="d21b86e6-517d-44f3-9422-5ba22f78afe8" containerName="revoke-keys" Jul 19 08:21:15.658452 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:15.658392 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fdbad2e2-795d7b46bb-24d7l" Jul 19 08:21:15.661161 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:15.661134 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-fdbad2e2-serving-cert\"" Jul 19 08:21:15.661699 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:15.661345 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 19 08:21:15.661699 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:15.661303 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 19 08:21:15.661699 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:15.661210 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-tdbkh\"" Jul 19 08:21:15.664037 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:15.664005 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-fdbad2e2-795d7b46bb-24d7l"] Jul 19 08:21:15.674299 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:15.674276 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:21:15.692274 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:15.692247 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ns74\" (UniqueName: \"kubernetes.io/projected/9d8a46d6-d163-4283-89a2-a50b1d40dbe5-kube-api-access-4ns74\") pod \"maas-api-e2e-derive-fdbad2e2-795d7b46bb-24d7l\" (UID: \"9d8a46d6-d163-4283-89a2-a50b1d40dbe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fdbad2e2-795d7b46bb-24d7l" Jul 19 08:21:15.692424 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:15.692361 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d8a46d6-d163-4283-89a2-a50b1d40dbe5-openshift-service-ca\") pod \"maas-api-e2e-derive-fdbad2e2-795d7b46bb-24d7l\" (UID: \"9d8a46d6-d163-4283-89a2-a50b1d40dbe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fdbad2e2-795d7b46bb-24d7l" Jul 19 08:21:15.692424 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:15.692407 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d8a46d6-d163-4283-89a2-a50b1d40dbe5-maas-api-tls\") pod \"maas-api-e2e-derive-fdbad2e2-795d7b46bb-24d7l\" (UID: \"9d8a46d6-d163-4283-89a2-a50b1d40dbe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fdbad2e2-795d7b46bb-24d7l" Jul 19 08:21:15.793628 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:15.793588 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ns74\" (UniqueName: \"kubernetes.io/projected/9d8a46d6-d163-4283-89a2-a50b1d40dbe5-kube-api-access-4ns74\") pod \"maas-api-e2e-derive-fdbad2e2-795d7b46bb-24d7l\" (UID: \"9d8a46d6-d163-4283-89a2-a50b1d40dbe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fdbad2e2-795d7b46bb-24d7l" Jul 19 08:21:15.793841 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:15.793693 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d8a46d6-d163-4283-89a2-a50b1d40dbe5-openshift-service-ca\") pod \"maas-api-e2e-derive-fdbad2e2-795d7b46bb-24d7l\" (UID: \"9d8a46d6-d163-4283-89a2-a50b1d40dbe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fdbad2e2-795d7b46bb-24d7l" Jul 19 08:21:15.793841 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:15.793731 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d8a46d6-d163-4283-89a2-a50b1d40dbe5-maas-api-tls\") pod \"maas-api-e2e-derive-fdbad2e2-795d7b46bb-24d7l\" (UID: \"9d8a46d6-d163-4283-89a2-a50b1d40dbe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fdbad2e2-795d7b46bb-24d7l" Jul 19 08:21:15.793943 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:21:15.793890 2574 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-fdbad2e2-serving-cert: secret "maas-api-e2e-derive-fdbad2e2-serving-cert" not found Jul 19 08:21:15.793984 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:21:15.793965 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d8a46d6-d163-4283-89a2-a50b1d40dbe5-maas-api-tls podName:9d8a46d6-d163-4283-89a2-a50b1d40dbe5 nodeName:}" failed. No retries permitted until 2026-07-19 08:21:16.293943872 +0000 UTC m=+2342.648253454 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9d8a46d6-d163-4283-89a2-a50b1d40dbe5-maas-api-tls") pod "maas-api-e2e-derive-fdbad2e2-795d7b46bb-24d7l" (UID: "9d8a46d6-d163-4283-89a2-a50b1d40dbe5") : secret "maas-api-e2e-derive-fdbad2e2-serving-cert" not found Jul 19 08:21:15.794378 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:15.794356 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d8a46d6-d163-4283-89a2-a50b1d40dbe5-openshift-service-ca\") pod \"maas-api-e2e-derive-fdbad2e2-795d7b46bb-24d7l\" (UID: \"9d8a46d6-d163-4283-89a2-a50b1d40dbe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fdbad2e2-795d7b46bb-24d7l" Jul 19 08:21:15.804225 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:15.804201 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ns74\" (UniqueName: \"kubernetes.io/projected/9d8a46d6-d163-4283-89a2-a50b1d40dbe5-kube-api-access-4ns74\") pod \"maas-api-e2e-derive-fdbad2e2-795d7b46bb-24d7l\" (UID: \"9d8a46d6-d163-4283-89a2-a50b1d40dbe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fdbad2e2-795d7b46bb-24d7l" Jul 19 08:21:16.123624 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:16.123582 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:21:16.298289 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:16.298251 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d8a46d6-d163-4283-89a2-a50b1d40dbe5-maas-api-tls\") pod \"maas-api-e2e-derive-fdbad2e2-795d7b46bb-24d7l\" (UID: \"9d8a46d6-d163-4283-89a2-a50b1d40dbe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fdbad2e2-795d7b46bb-24d7l" Jul 19 08:21:16.300973 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:16.300944 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d8a46d6-d163-4283-89a2-a50b1d40dbe5-maas-api-tls\") pod \"maas-api-e2e-derive-fdbad2e2-795d7b46bb-24d7l\" (UID: \"9d8a46d6-d163-4283-89a2-a50b1d40dbe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fdbad2e2-795d7b46bb-24d7l" Jul 19 08:21:16.328250 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:16.328218 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:21:16.425704 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:16.425617 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:21:16.562535 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:16.562493 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-fdbad2e2-openshift-default-5cf7ff57b7-g8sqd" Jul 19 08:21:16.563627 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:16.563602 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-fdbad2e2-openshift-default-5cf7ff57b7-g8sqd" Jul 19 08:21:16.572571 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:16.572551 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fdbad2e2-795d7b46bb-24d7l" Jul 19 08:21:16.719744 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:16.719404 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-fdbad2e2-795d7b46bb-24d7l"] Jul 19 08:21:16.722433 ip-10-0-136-70 kubenswrapper[2574]: W0719 08:21:16.722302 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d8a46d6_d163_4283_89a2_a50b1d40dbe5.slice/crio-655d8e4206338ae78d44f943e4f93609cf15500ec19625ad236c411e81a70731 WatchSource:0}: Error finding container 655d8e4206338ae78d44f943e4f93609cf15500ec19625ad236c411e81a70731: Status 404 returned error can't find the container with id 655d8e4206338ae78d44f943e4f93609cf15500ec19625ad236c411e81a70731 Jul 19 08:21:17.569107 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:17.569073 2574 generic.go:356] "Generic (PLEG): container finished" podID="9d8a46d6-d163-4283-89a2-a50b1d40dbe5" containerID="ec2b5adf9bd1585ad453949e78d22317ed7600b7b4233709cd448882843012bb" exitCode=1 Jul 19 08:21:17.569612 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:17.569153 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fdbad2e2-795d7b46bb-24d7l" event={"ID":"9d8a46d6-d163-4283-89a2-a50b1d40dbe5","Type":"ContainerDied","Data":"ec2b5adf9bd1585ad453949e78d22317ed7600b7b4233709cd448882843012bb"} Jul 19 08:21:17.569612 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:17.569207 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fdbad2e2-795d7b46bb-24d7l" event={"ID":"9d8a46d6-d163-4283-89a2-a50b1d40dbe5","Type":"ContainerStarted","Data":"655d8e4206338ae78d44f943e4f93609cf15500ec19625ad236c411e81a70731"} Jul 19 08:21:17.569612 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:17.569443 2574 scope.go:117] "RemoveContainer" containerID="ec2b5adf9bd1585ad453949e78d22317ed7600b7b4233709cd448882843012bb" Jul 19 08:21:18.480102 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:18.480063 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 08:21:18.575289 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:18.575253 2574 generic.go:356] "Generic (PLEG): container finished" podID="9d8a46d6-d163-4283-89a2-a50b1d40dbe5" containerID="1424dfec2ee86560d78ce2272f9c322b12c6a74384091206e221104ab2d6841f" exitCode=1 Jul 19 08:21:18.575759 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:18.575341 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fdbad2e2-795d7b46bb-24d7l" event={"ID":"9d8a46d6-d163-4283-89a2-a50b1d40dbe5","Type":"ContainerDied","Data":"1424dfec2ee86560d78ce2272f9c322b12c6a74384091206e221104ab2d6841f"} Jul 19 08:21:18.575759 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:18.575395 2574 scope.go:117] "RemoveContainer" containerID="ec2b5adf9bd1585ad453949e78d22317ed7600b7b4233709cd448882843012bb" Jul 19 08:21:18.575759 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:18.575705 2574 scope.go:117] "RemoveContainer" containerID="1424dfec2ee86560d78ce2272f9c322b12c6a74384091206e221104ab2d6841f" Jul 19 08:21:18.575953 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:21:18.575933 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-derive-fdbad2e2-795d7b46bb-24d7l_odh-ai-gateway-infra(9d8a46d6-d163-4283-89a2-a50b1d40dbe5)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fdbad2e2-795d7b46bb-24d7l" podUID="9d8a46d6-d163-4283-89a2-a50b1d40dbe5" Jul 19 08:21:19.585448 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:19.585412 2574 scope.go:117] "RemoveContainer" containerID="1424dfec2ee86560d78ce2272f9c322b12c6a74384091206e221104ab2d6841f" Jul 19 08:21:19.585885 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:21:19.585674 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-derive-fdbad2e2-795d7b46bb-24d7l_odh-ai-gateway-infra(9d8a46d6-d163-4283-89a2-a50b1d40dbe5)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fdbad2e2-795d7b46bb-24d7l" podUID="9d8a46d6-d163-4283-89a2-a50b1d40dbe5" Jul 19 08:21:21.573437 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:21.573397 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fdbad2e2-795d7b46bb-24d7l" Jul 19 08:21:21.573886 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:21.573840 2574 scope.go:117] "RemoveContainer" containerID="1424dfec2ee86560d78ce2272f9c322b12c6a74384091206e221104ab2d6841f" Jul 19 08:21:21.574020 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:21:21.574000 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-derive-fdbad2e2-795d7b46bb-24d7l_odh-ai-gateway-infra(9d8a46d6-d163-4283-89a2-a50b1d40dbe5)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fdbad2e2-795d7b46bb-24d7l" podUID="9d8a46d6-d163-4283-89a2-a50b1d40dbe5" Jul 19 08:21:26.573667 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:26.573573 2574 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fdbad2e2-795d7b46bb-24d7l" Jul 19 08:21:26.574079 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:26.573960 2574 scope.go:117] "RemoveContainer" containerID="1424dfec2ee86560d78ce2272f9c322b12c6a74384091206e221104ab2d6841f" Jul 19 08:21:26.574152 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:21:26.574132 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-derive-fdbad2e2-795d7b46bb-24d7l_odh-ai-gateway-infra(9d8a46d6-d163-4283-89a2-a50b1d40dbe5)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fdbad2e2-795d7b46bb-24d7l" podUID="9d8a46d6-d163-4283-89a2-a50b1d40dbe5" Jul 19 08:21:26.622359 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:26.622322 2574 generic.go:356] "Generic (PLEG): container finished" podID="dac3f27a-2ab3-4707-84b9-ed590c895b2c" containerID="50b6323edf7d51394e4972c91bf1408dd9cb1ed8755c0a18475bf7f91fe6ed14" exitCode=0 Jul 19 08:21:26.622554 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:26.622396 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-fdbad2e2-openshift-default-5cf7ff57b7-g8sqd" event={"ID":"dac3f27a-2ab3-4707-84b9-ed590c895b2c","Type":"ContainerDied","Data":"50b6323edf7d51394e4972c91bf1408dd9cb1ed8755c0a18475bf7f91fe6ed14"} Jul 19 08:21:26.622895 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:26.622880 2574 scope.go:117] "RemoveContainer" containerID="50b6323edf7d51394e4972c91bf1408dd9cb1ed8755c0a18475bf7f91fe6ed14" Jul 19 08:21:26.623968 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:26.623936 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 19 08:21:26.624088 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:26.624026 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 19 08:21:26.624088 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:26.624058 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 19 08:21:27.633498 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:27.633448 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-fdbad2e2-openshift-default-5cf7ff57b7-g8sqd" event={"ID":"dac3f27a-2ab3-4707-84b9-ed590c895b2c","Type":"ContainerStarted","Data":"e8fbb04e8816d1daec868eadab04e8ee9f7125ec347930ff2c232c544e3dbeeb"} Jul 19 08:21:28.588509 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:28.588471 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-fdbad2e2-openshift-default-5cf7ff57b7-g8sqd" Jul 19 08:21:28.589759 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:28.589732 2574 patch_prober.go:28] interesting pod/e2e-derive-fdbad2e2-openshift-default-5cf7ff57b7-g8sqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Jul 19 08:21:28.589882 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:28.589784 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-fdbad2e2-openshift-default-5cf7ff57b7-g8sqd" podUID="dac3f27a-2ab3-4707-84b9-ed590c895b2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Jul 19 08:21:29.588670 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:29.588630 2574 patch_prober.go:28] interesting pod/e2e-derive-fdbad2e2-openshift-default-5cf7ff57b7-g8sqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Jul 19 08:21:29.589083 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:29.588688 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-fdbad2e2-openshift-default-5cf7ff57b7-g8sqd" podUID="dac3f27a-2ab3-4707-84b9-ed590c895b2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Jul 19 08:21:30.589187 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:30.589146 2574 patch_prober.go:28] interesting pod/e2e-derive-fdbad2e2-openshift-default-5cf7ff57b7-g8sqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Jul 19 08:21:30.589622 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:30.589213 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-fdbad2e2-openshift-default-5cf7ff57b7-g8sqd" podUID="dac3f27a-2ab3-4707-84b9-ed590c895b2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Jul 19 08:21:31.562931 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:31.562893 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-fdbad2e2-openshift-default-5cf7ff57b7-g8sqd" Jul 19 08:21:31.593369 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:31.593339 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-fdbad2e2-openshift-default-5cf7ff57b7-g8sqd" Jul 19 08:21:31.661208 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:31.661177 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-fdbad2e2-openshift-default-5cf7ff57b7-g8sqd" Jul 19 08:21:34.495819 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:34.495780 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:21:34.545026 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:34.544985 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:21:37.943764 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:37.943733 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:21:40.553120 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:40.553085 2574 scope.go:117] "RemoveContainer" containerID="1424dfec2ee86560d78ce2272f9c322b12c6a74384091206e221104ab2d6841f" Jul 19 08:21:41.714096 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:41.714058 2574 generic.go:356] "Generic (PLEG): container finished" podID="9d8a46d6-d163-4283-89a2-a50b1d40dbe5" containerID="d6dbd3686c32a6c6b0cedec42e14b8f9bd689b5e94b3f96abee83a5a9007e8c9" exitCode=1 Jul 19 08:21:41.714571 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:41.714137 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fdbad2e2-795d7b46bb-24d7l" event={"ID":"9d8a46d6-d163-4283-89a2-a50b1d40dbe5","Type":"ContainerDied","Data":"d6dbd3686c32a6c6b0cedec42e14b8f9bd689b5e94b3f96abee83a5a9007e8c9"} Jul 19 08:21:41.714571 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:41.714214 2574 scope.go:117] "RemoveContainer" containerID="1424dfec2ee86560d78ce2272f9c322b12c6a74384091206e221104ab2d6841f" Jul 19 08:21:41.714710 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:41.714571 2574 scope.go:117] "RemoveContainer" containerID="d6dbd3686c32a6c6b0cedec42e14b8f9bd689b5e94b3f96abee83a5a9007e8c9" Jul 19 08:21:41.714816 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:21:41.714796 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-derive-fdbad2e2-795d7b46bb-24d7l_odh-ai-gateway-infra(9d8a46d6-d163-4283-89a2-a50b1d40dbe5)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fdbad2e2-795d7b46bb-24d7l" podUID="9d8a46d6-d163-4283-89a2-a50b1d40dbe5" Jul 19 08:21:46.573017 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:46.572982 2574 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fdbad2e2-795d7b46bb-24d7l" Jul 19 08:21:46.573433 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:46.573041 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fdbad2e2-795d7b46bb-24d7l" Jul 19 08:21:46.573433 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:46.573408 2574 scope.go:117] "RemoveContainer" containerID="d6dbd3686c32a6c6b0cedec42e14b8f9bd689b5e94b3f96abee83a5a9007e8c9" Jul 19 08:21:46.573670 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:21:46.573648 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-derive-fdbad2e2-795d7b46bb-24d7l_odh-ai-gateway-infra(9d8a46d6-d163-4283-89a2-a50b1d40dbe5)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fdbad2e2-795d7b46bb-24d7l" podUID="9d8a46d6-d163-4283-89a2-a50b1d40dbe5" Jul 19 08:21:58.552619 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:21:58.552583 2574 scope.go:117] "RemoveContainer" containerID="d6dbd3686c32a6c6b0cedec42e14b8f9bd689b5e94b3f96abee83a5a9007e8c9" Jul 19 08:21:58.553103 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:21:58.552780 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-derive-fdbad2e2-795d7b46bb-24d7l_odh-ai-gateway-infra(9d8a46d6-d163-4283-89a2-a50b1d40dbe5)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fdbad2e2-795d7b46bb-24d7l" podUID="9d8a46d6-d163-4283-89a2-a50b1d40dbe5" Jul 19 08:22:12.553074 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:22:12.553034 2574 scope.go:117] "RemoveContainer" containerID="d6dbd3686c32a6c6b0cedec42e14b8f9bd689b5e94b3f96abee83a5a9007e8c9" Jul 19 08:22:13.875512 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:22:13.875472 2574 generic.go:356] "Generic (PLEG): container finished" podID="9d8a46d6-d163-4283-89a2-a50b1d40dbe5" containerID="144e6d6683ec6e5c9fd46728bdbe0bfe1e665b92f91aabb5a07287e2aa9c35ce" exitCode=1 Jul 19 08:22:13.875953 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:22:13.875551 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fdbad2e2-795d7b46bb-24d7l" event={"ID":"9d8a46d6-d163-4283-89a2-a50b1d40dbe5","Type":"ContainerDied","Data":"144e6d6683ec6e5c9fd46728bdbe0bfe1e665b92f91aabb5a07287e2aa9c35ce"} Jul 19 08:22:13.875953 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:22:13.875631 2574 scope.go:117] "RemoveContainer" containerID="d6dbd3686c32a6c6b0cedec42e14b8f9bd689b5e94b3f96abee83a5a9007e8c9" Jul 19 08:22:13.876079 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:22:13.876039 2574 scope.go:117] "RemoveContainer" containerID="144e6d6683ec6e5c9fd46728bdbe0bfe1e665b92f91aabb5a07287e2aa9c35ce" Jul 19 08:22:13.876316 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:22:13.876292 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-derive-fdbad2e2-795d7b46bb-24d7l_odh-ai-gateway-infra(9d8a46d6-d163-4283-89a2-a50b1d40dbe5)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fdbad2e2-795d7b46bb-24d7l" podUID="9d8a46d6-d163-4283-89a2-a50b1d40dbe5" Jul 19 08:22:14.692431 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:22:14.692399 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wmhfd_dd68d31d-f763-433a-867b-65b12eabac76/console-operator/1.log" Jul 19 08:22:14.696273 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:22:14.696251 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wmhfd_dd68d31d-f763-433a-867b-65b12eabac76/console-operator/1.log" Jul 19 08:22:16.572788 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:22:16.572754 2574 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fdbad2e2-795d7b46bb-24d7l" Jul 19 08:22:16.573216 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:22:16.572807 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fdbad2e2-795d7b46bb-24d7l" Jul 19 08:22:16.573216 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:22:16.573132 2574 scope.go:117] "RemoveContainer" containerID="144e6d6683ec6e5c9fd46728bdbe0bfe1e665b92f91aabb5a07287e2aa9c35ce" Jul 19 08:22:16.573316 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:22:16.573295 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-derive-fdbad2e2-795d7b46bb-24d7l_odh-ai-gateway-infra(9d8a46d6-d163-4283-89a2-a50b1d40dbe5)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fdbad2e2-795d7b46bb-24d7l" podUID="9d8a46d6-d163-4283-89a2-a50b1d40dbe5" Jul 19 08:22:29.553281 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:22:29.553244 2574 scope.go:117] "RemoveContainer" containerID="144e6d6683ec6e5c9fd46728bdbe0bfe1e665b92f91aabb5a07287e2aa9c35ce" Jul 19 08:22:29.553724 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:22:29.553438 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-derive-fdbad2e2-795d7b46bb-24d7l_odh-ai-gateway-infra(9d8a46d6-d163-4283-89a2-a50b1d40dbe5)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fdbad2e2-795d7b46bb-24d7l" podUID="9d8a46d6-d163-4283-89a2-a50b1d40dbe5" Jul 19 08:22:40.552769 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:22:40.552687 2574 scope.go:117] "RemoveContainer" containerID="144e6d6683ec6e5c9fd46728bdbe0bfe1e665b92f91aabb5a07287e2aa9c35ce" Jul 19 08:22:40.553246 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:22:40.552855 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-derive-fdbad2e2-795d7b46bb-24d7l_odh-ai-gateway-infra(9d8a46d6-d163-4283-89a2-a50b1d40dbe5)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fdbad2e2-795d7b46bb-24d7l" podUID="9d8a46d6-d163-4283-89a2-a50b1d40dbe5" Jul 19 08:22:52.552738 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:22:52.552704 2574 scope.go:117] "RemoveContainer" containerID="144e6d6683ec6e5c9fd46728bdbe0bfe1e665b92f91aabb5a07287e2aa9c35ce" Jul 19 08:22:52.553154 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:22:52.552923 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-derive-fdbad2e2-795d7b46bb-24d7l_odh-ai-gateway-infra(9d8a46d6-d163-4283-89a2-a50b1d40dbe5)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fdbad2e2-795d7b46bb-24d7l" podUID="9d8a46d6-d163-4283-89a2-a50b1d40dbe5" Jul 19 08:23:05.553284 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:23:05.553248 2574 scope.go:117] "RemoveContainer" containerID="144e6d6683ec6e5c9fd46728bdbe0bfe1e665b92f91aabb5a07287e2aa9c35ce" Jul 19 08:23:06.136736 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:23:06.136695 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fdbad2e2-795d7b46bb-24d7l" event={"ID":"9d8a46d6-d163-4283-89a2-a50b1d40dbe5","Type":"ContainerStarted","Data":"15dad8837efcd969e12013531b57346d6d2437764273af951286e1dbcf72c9c4"} Jul 19 08:23:06.136976 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:23:06.136957 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fdbad2e2-795d7b46bb-24d7l" Jul 19 08:23:06.153719 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:23:06.153666 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fdbad2e2-795d7b46bb-24d7l" podStartSLOduration=109.000266111 podStartE2EDuration="1m51.153648721s" totalImagesPullingTime="2.15338261s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 08:21:15 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:23:06.152238508 +0000 UTC m=+2452.506548110" watchObservedRunningTime="2026-07-19 08:23:06.153648721 +0000 UTC m=+2452.507958322" Jul 19 08:23:07.143705 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:23:07.143665 2574 generic.go:356] "Generic (PLEG): container finished" podID="9d8a46d6-d163-4283-89a2-a50b1d40dbe5" containerID="15dad8837efcd969e12013531b57346d6d2437764273af951286e1dbcf72c9c4" exitCode=1 Jul 19 08:23:07.144209 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:23:07.143746 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fdbad2e2-795d7b46bb-24d7l" event={"ID":"9d8a46d6-d163-4283-89a2-a50b1d40dbe5","Type":"ContainerDied","Data":"15dad8837efcd969e12013531b57346d6d2437764273af951286e1dbcf72c9c4"} Jul 19 08:23:07.144209 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:23:07.143809 2574 scope.go:117] "RemoveContainer" containerID="144e6d6683ec6e5c9fd46728bdbe0bfe1e665b92f91aabb5a07287e2aa9c35ce" Jul 19 08:23:07.144209 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:23:07.143965 2574 scope.go:117] "RemoveContainer" containerID="15dad8837efcd969e12013531b57346d6d2437764273af951286e1dbcf72c9c4" Jul 19 08:23:07.144339 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:23:07.144244 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-fdbad2e2-795d7b46bb-24d7l_odh-ai-gateway-infra(9d8a46d6-d163-4283-89a2-a50b1d40dbe5)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fdbad2e2-795d7b46bb-24d7l" podUID="9d8a46d6-d163-4283-89a2-a50b1d40dbe5" Jul 19 08:23:08.152038 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:23:08.151995 2574 scope.go:117] "RemoveContainer" containerID="15dad8837efcd969e12013531b57346d6d2437764273af951286e1dbcf72c9c4" Jul 19 08:23:08.152444 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:23:08.152204 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-fdbad2e2-795d7b46bb-24d7l_odh-ai-gateway-infra(9d8a46d6-d163-4283-89a2-a50b1d40dbe5)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fdbad2e2-795d7b46bb-24d7l" podUID="9d8a46d6-d163-4283-89a2-a50b1d40dbe5" Jul 19 08:23:16.572753 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:23:16.572717 2574 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fdbad2e2-795d7b46bb-24d7l" Jul 19 08:23:16.573354 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:23:16.573195 2574 scope.go:117] "RemoveContainer" containerID="15dad8837efcd969e12013531b57346d6d2437764273af951286e1dbcf72c9c4" Jul 19 08:23:16.573454 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:23:16.573427 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-fdbad2e2-795d7b46bb-24d7l_odh-ai-gateway-infra(9d8a46d6-d163-4283-89a2-a50b1d40dbe5)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fdbad2e2-795d7b46bb-24d7l" podUID="9d8a46d6-d163-4283-89a2-a50b1d40dbe5" Jul 19 08:23:27.553192 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:23:27.553158 2574 scope.go:117] "RemoveContainer" containerID="15dad8837efcd969e12013531b57346d6d2437764273af951286e1dbcf72c9c4" Jul 19 08:23:27.553648 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:23:27.553343 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-fdbad2e2-795d7b46bb-24d7l_odh-ai-gateway-infra(9d8a46d6-d163-4283-89a2-a50b1d40dbe5)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fdbad2e2-795d7b46bb-24d7l" podUID="9d8a46d6-d163-4283-89a2-a50b1d40dbe5" Jul 19 08:23:38.558790 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:23:38.558758 2574 scope.go:117] "RemoveContainer" containerID="15dad8837efcd969e12013531b57346d6d2437764273af951286e1dbcf72c9c4" Jul 19 08:23:38.559185 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:23:38.558949 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-fdbad2e2-795d7b46bb-24d7l_odh-ai-gateway-infra(9d8a46d6-d163-4283-89a2-a50b1d40dbe5)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fdbad2e2-795d7b46bb-24d7l" podUID="9d8a46d6-d163-4283-89a2-a50b1d40dbe5" Jul 19 08:23:49.553119 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:23:49.553084 2574 scope.go:117] "RemoveContainer" containerID="15dad8837efcd969e12013531b57346d6d2437764273af951286e1dbcf72c9c4" Jul 19 08:23:49.553642 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:23:49.553279 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-fdbad2e2-795d7b46bb-24d7l_odh-ai-gateway-infra(9d8a46d6-d163-4283-89a2-a50b1d40dbe5)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fdbad2e2-795d7b46bb-24d7l" podUID="9d8a46d6-d163-4283-89a2-a50b1d40dbe5" Jul 19 08:23:51.085670 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:23:51.085630 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:23:51.169475 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:23:51.169433 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:24:00.553654 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:24:00.553615 2574 scope.go:117] "RemoveContainer" containerID="15dad8837efcd969e12013531b57346d6d2437764273af951286e1dbcf72c9c4" Jul 19 08:24:00.554196 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:24:00.553783 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-fdbad2e2-795d7b46bb-24d7l_odh-ai-gateway-infra(9d8a46d6-d163-4283-89a2-a50b1d40dbe5)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fdbad2e2-795d7b46bb-24d7l" podUID="9d8a46d6-d163-4283-89a2-a50b1d40dbe5" Jul 19 08:24:00.579429 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:24:00.579382 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:24:02.046319 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:24:02.046283 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:24:12.553069 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:24:12.552978 2574 scope.go:117] "RemoveContainer" containerID="15dad8837efcd969e12013531b57346d6d2437764273af951286e1dbcf72c9c4" Jul 19 08:24:12.553595 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:24:12.553165 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-fdbad2e2-795d7b46bb-24d7l_odh-ai-gateway-infra(9d8a46d6-d163-4283-89a2-a50b1d40dbe5)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fdbad2e2-795d7b46bb-24d7l" podUID="9d8a46d6-d163-4283-89a2-a50b1d40dbe5" Jul 19 08:24:16.447775 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:24:16.447731 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:24:19.932919 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:24:19.932878 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-fdbad2e2-openshift-default-5cf7ff57b7-g8sqd"] Jul 19 08:24:19.933327 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:24:19.933113 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-fdbad2e2-openshift-default-5cf7ff57b7-g8sqd" podUID="dac3f27a-2ab3-4707-84b9-ed590c895b2c" containerName="istio-proxy" containerID="cri-o://e8fbb04e8816d1daec868eadab04e8ee9f7125ec347930ff2c232c544e3dbeeb" gracePeriod=30 Jul 19 08:24:19.956972 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:24:19.956934 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:24:20.059563 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:24:20.059504 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:24:22.776203 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:24:22.776169 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 08:24:23.552616 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:24:23.552581 2574 scope.go:117] "RemoveContainer" containerID="15dad8837efcd969e12013531b57346d6d2437764273af951286e1dbcf72c9c4" Jul 19 08:24:23.552807 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:24:23.552783 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-fdbad2e2-795d7b46bb-24d7l_odh-ai-gateway-infra(9d8a46d6-d163-4283-89a2-a50b1d40dbe5)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fdbad2e2-795d7b46bb-24d7l" podUID="9d8a46d6-d163-4283-89a2-a50b1d40dbe5" Jul 19 08:24:25.186799 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:24:25.186774 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-fdbad2e2-openshift-default-5cf7ff57b7-g8sqd" Jul 19 08:24:25.258956 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:24:25.258925 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dac3f27a-2ab3-4707-84b9-ed590c895b2c-istio-podinfo\") pod \"dac3f27a-2ab3-4707-84b9-ed590c895b2c\" (UID: \"dac3f27a-2ab3-4707-84b9-ed590c895b2c\") " Jul 19 08:24:25.259139 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:24:25.258973 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dac3f27a-2ab3-4707-84b9-ed590c895b2c-workload-socket\") pod \"dac3f27a-2ab3-4707-84b9-ed590c895b2c\" (UID: \"dac3f27a-2ab3-4707-84b9-ed590c895b2c\") " Jul 19 08:24:25.259139 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:24:25.258997 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dac3f27a-2ab3-4707-84b9-ed590c895b2c-credential-socket\") pod \"dac3f27a-2ab3-4707-84b9-ed590c895b2c\" (UID: \"dac3f27a-2ab3-4707-84b9-ed590c895b2c\") " Jul 19 08:24:25.259139 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:24:25.259027 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dac3f27a-2ab3-4707-84b9-ed590c895b2c-istio-token\") pod \"dac3f27a-2ab3-4707-84b9-ed590c895b2c\" (UID: \"dac3f27a-2ab3-4707-84b9-ed590c895b2c\") " Jul 19 08:24:25.259139 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:24:25.259079 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9rw9m\" (UniqueName: \"kubernetes.io/projected/dac3f27a-2ab3-4707-84b9-ed590c895b2c-kube-api-access-9rw9m\") pod \"dac3f27a-2ab3-4707-84b9-ed590c895b2c\" (UID: \"dac3f27a-2ab3-4707-84b9-ed590c895b2c\") " Jul 19 08:24:25.259139 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:24:25.259107 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dac3f27a-2ab3-4707-84b9-ed590c895b2c-istiod-ca-cert\") pod \"dac3f27a-2ab3-4707-84b9-ed590c895b2c\" (UID: \"dac3f27a-2ab3-4707-84b9-ed590c895b2c\") " Jul 19 08:24:25.259401 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:24:25.259148 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dac3f27a-2ab3-4707-84b9-ed590c895b2c-istio-data\") pod \"dac3f27a-2ab3-4707-84b9-ed590c895b2c\" (UID: \"dac3f27a-2ab3-4707-84b9-ed590c895b2c\") " Jul 19 08:24:25.259401 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:24:25.259335 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dac3f27a-2ab3-4707-84b9-ed590c895b2c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "dac3f27a-2ab3-4707-84b9-ed590c895b2c" (UID: "dac3f27a-2ab3-4707-84b9-ed590c895b2c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 08:24:25.259401 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:24:25.259343 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dac3f27a-2ab3-4707-84b9-ed590c895b2c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "dac3f27a-2ab3-4707-84b9-ed590c895b2c" (UID: "dac3f27a-2ab3-4707-84b9-ed590c895b2c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 08:24:25.259737 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:24:25.259717 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dac3f27a-2ab3-4707-84b9-ed590c895b2c-istio-envoy\") pod \"dac3f27a-2ab3-4707-84b9-ed590c895b2c\" (UID: \"dac3f27a-2ab3-4707-84b9-ed590c895b2c\") " Jul 19 08:24:25.259926 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:24:25.259900 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dac3f27a-2ab3-4707-84b9-ed590c895b2c-workload-certs\") pod \"dac3f27a-2ab3-4707-84b9-ed590c895b2c\" (UID: \"dac3f27a-2ab3-4707-84b9-ed590c895b2c\") " Jul 19 08:24:25.260320 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:24:25.259752 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dac3f27a-2ab3-4707-84b9-ed590c895b2c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "dac3f27a-2ab3-4707-84b9-ed590c895b2c" (UID: "dac3f27a-2ab3-4707-84b9-ed590c895b2c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 08:24:25.260320 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:24:25.260211 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dac3f27a-2ab3-4707-84b9-ed590c895b2c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "dac3f27a-2ab3-4707-84b9-ed590c895b2c" (UID: "dac3f27a-2ab3-4707-84b9-ed590c895b2c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 08:24:25.260482 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:24:25.260462 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dac3f27a-2ab3-4707-84b9-ed590c895b2c-workload-certs\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:24:25.260566 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:24:25.260481 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dac3f27a-2ab3-4707-84b9-ed590c895b2c-workload-socket\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:24:25.260566 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:24:25.260496 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dac3f27a-2ab3-4707-84b9-ed590c895b2c-credential-socket\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:24:25.260566 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:24:25.260510 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dac3f27a-2ab3-4707-84b9-ed590c895b2c-istiod-ca-cert\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:24:25.260685 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:24:25.260620 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dac3f27a-2ab3-4707-84b9-ed590c895b2c-istio-data" (OuterVolumeSpecName: "istio-data") pod "dac3f27a-2ab3-4707-84b9-ed590c895b2c" (UID: "dac3f27a-2ab3-4707-84b9-ed590c895b2c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 08:24:25.261555 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:24:25.261500 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dac3f27a-2ab3-4707-84b9-ed590c895b2c-kube-api-access-9rw9m" (OuterVolumeSpecName: "kube-api-access-9rw9m") pod "dac3f27a-2ab3-4707-84b9-ed590c895b2c" (UID: "dac3f27a-2ab3-4707-84b9-ed590c895b2c"). InnerVolumeSpecName "kube-api-access-9rw9m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 08:24:25.261708 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:24:25.261686 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dac3f27a-2ab3-4707-84b9-ed590c895b2c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "dac3f27a-2ab3-4707-84b9-ed590c895b2c" (UID: "dac3f27a-2ab3-4707-84b9-ed590c895b2c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 19 08:24:25.261762 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:24:25.261694 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dac3f27a-2ab3-4707-84b9-ed590c895b2c-istio-token" (OuterVolumeSpecName: "istio-token") pod "dac3f27a-2ab3-4707-84b9-ed590c895b2c" (UID: "dac3f27a-2ab3-4707-84b9-ed590c895b2c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 08:24:25.262057 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:24:25.262037 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dac3f27a-2ab3-4707-84b9-ed590c895b2c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "dac3f27a-2ab3-4707-84b9-ed590c895b2c" (UID: "dac3f27a-2ab3-4707-84b9-ed590c895b2c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 08:24:25.361994 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:24:25.361954 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dac3f27a-2ab3-4707-84b9-ed590c895b2c-istio-envoy\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:24:25.361994 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:24:25.361987 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dac3f27a-2ab3-4707-84b9-ed590c895b2c-istio-podinfo\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:24:25.361994 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:24:25.361999 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dac3f27a-2ab3-4707-84b9-ed590c895b2c-istio-token\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:24:25.362250 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:24:25.362011 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rw9m\" (UniqueName: \"kubernetes.io/projected/dac3f27a-2ab3-4707-84b9-ed590c895b2c-kube-api-access-9rw9m\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:24:25.362250 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:24:25.362024 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dac3f27a-2ab3-4707-84b9-ed590c895b2c-istio-data\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:24:25.564806 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:24:25.564651 2574 generic.go:356] "Generic (PLEG): container finished" podID="dac3f27a-2ab3-4707-84b9-ed590c895b2c" containerID="e8fbb04e8816d1daec868eadab04e8ee9f7125ec347930ff2c232c544e3dbeeb" exitCode=0 Jul 19 08:24:25.564806 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:24:25.564723 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-fdbad2e2-openshift-default-5cf7ff57b7-g8sqd" Jul 19 08:24:25.564806 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:24:25.564719 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-fdbad2e2-openshift-default-5cf7ff57b7-g8sqd" event={"ID":"dac3f27a-2ab3-4707-84b9-ed590c895b2c","Type":"ContainerDied","Data":"e8fbb04e8816d1daec868eadab04e8ee9f7125ec347930ff2c232c544e3dbeeb"} Jul 19 08:24:25.565030 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:24:25.564829 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-fdbad2e2-openshift-default-5cf7ff57b7-g8sqd" event={"ID":"dac3f27a-2ab3-4707-84b9-ed590c895b2c","Type":"ContainerDied","Data":"bf797a3d58e84513c4d0ca2405313972ff5431512d5e7c23c00f81989dd44513"} Jul 19 08:24:25.565030 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:24:25.564874 2574 scope.go:117] "RemoveContainer" containerID="e8fbb04e8816d1daec868eadab04e8ee9f7125ec347930ff2c232c544e3dbeeb" Jul 19 08:24:25.569300 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:24:25.569274 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-fdbad2e2-795d7b46bb-24d7l"] Jul 19 08:24:25.578974 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:24:25.578952 2574 scope.go:117] "RemoveContainer" containerID="50b6323edf7d51394e4972c91bf1408dd9cb1ed8755c0a18475bf7f91fe6ed14" Jul 19 08:24:25.588084 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:24:25.588061 2574 scope.go:117] "RemoveContainer" containerID="e8fbb04e8816d1daec868eadab04e8ee9f7125ec347930ff2c232c544e3dbeeb" Jul 19 08:24:25.588360 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:24:25.588337 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8fbb04e8816d1daec868eadab04e8ee9f7125ec347930ff2c232c544e3dbeeb\": container with ID starting with e8fbb04e8816d1daec868eadab04e8ee9f7125ec347930ff2c232c544e3dbeeb not found: ID does not exist" containerID="e8fbb04e8816d1daec868eadab04e8ee9f7125ec347930ff2c232c544e3dbeeb" Jul 19 08:24:25.588433 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:24:25.588369 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8fbb04e8816d1daec868eadab04e8ee9f7125ec347930ff2c232c544e3dbeeb"} err="failed to get container status \"e8fbb04e8816d1daec868eadab04e8ee9f7125ec347930ff2c232c544e3dbeeb\": rpc error: code = NotFound desc = could not find container \"e8fbb04e8816d1daec868eadab04e8ee9f7125ec347930ff2c232c544e3dbeeb\": container with ID starting with e8fbb04e8816d1daec868eadab04e8ee9f7125ec347930ff2c232c544e3dbeeb not found: ID does not exist" Jul 19 08:24:25.588433 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:24:25.588389 2574 scope.go:117] "RemoveContainer" containerID="50b6323edf7d51394e4972c91bf1408dd9cb1ed8755c0a18475bf7f91fe6ed14" Jul 19 08:24:25.588676 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:24:25.588659 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50b6323edf7d51394e4972c91bf1408dd9cb1ed8755c0a18475bf7f91fe6ed14\": container with ID starting with 50b6323edf7d51394e4972c91bf1408dd9cb1ed8755c0a18475bf7f91fe6ed14 not found: ID does not exist" containerID="50b6323edf7d51394e4972c91bf1408dd9cb1ed8755c0a18475bf7f91fe6ed14" Jul 19 08:24:25.588733 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:24:25.588682 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50b6323edf7d51394e4972c91bf1408dd9cb1ed8755c0a18475bf7f91fe6ed14"} err="failed to get container status \"50b6323edf7d51394e4972c91bf1408dd9cb1ed8755c0a18475bf7f91fe6ed14\": rpc error: code = NotFound desc = could not find container \"50b6323edf7d51394e4972c91bf1408dd9cb1ed8755c0a18475bf7f91fe6ed14\": container with ID starting with 50b6323edf7d51394e4972c91bf1408dd9cb1ed8755c0a18475bf7f91fe6ed14 not found: ID does not exist" Jul 19 08:24:25.591158 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:24:25.591124 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:24:25.600198 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:24:25.600159 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-fdbad2e2-openshift-default-5cf7ff57b7-g8sqd"] Jul 19 08:24:25.607004 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:24:25.606982 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-fdbad2e2-openshift-default-5cf7ff57b7-g8sqd"] Jul 19 08:24:25.699673 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:24:25.699648 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fdbad2e2-795d7b46bb-24d7l" Jul 19 08:24:25.764816 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:24:25.764783 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4ns74\" (UniqueName: \"kubernetes.io/projected/9d8a46d6-d163-4283-89a2-a50b1d40dbe5-kube-api-access-4ns74\") pod \"9d8a46d6-d163-4283-89a2-a50b1d40dbe5\" (UID: \"9d8a46d6-d163-4283-89a2-a50b1d40dbe5\") " Jul 19 08:24:25.765003 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:24:25.764886 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d8a46d6-d163-4283-89a2-a50b1d40dbe5-openshift-service-ca\") pod \"9d8a46d6-d163-4283-89a2-a50b1d40dbe5\" (UID: \"9d8a46d6-d163-4283-89a2-a50b1d40dbe5\") " Jul 19 08:24:25.765003 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:24:25.764948 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d8a46d6-d163-4283-89a2-a50b1d40dbe5-maas-api-tls\") pod \"9d8a46d6-d163-4283-89a2-a50b1d40dbe5\" (UID: \"9d8a46d6-d163-4283-89a2-a50b1d40dbe5\") " Jul 19 08:24:25.765287 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:24:25.765263 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d8a46d6-d163-4283-89a2-a50b1d40dbe5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9d8a46d6-d163-4283-89a2-a50b1d40dbe5" (UID: "9d8a46d6-d163-4283-89a2-a50b1d40dbe5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 08:24:25.766922 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:24:25.766900 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d8a46d6-d163-4283-89a2-a50b1d40dbe5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9d8a46d6-d163-4283-89a2-a50b1d40dbe5" (UID: "9d8a46d6-d163-4283-89a2-a50b1d40dbe5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 08:24:25.766996 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:24:25.766948 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d8a46d6-d163-4283-89a2-a50b1d40dbe5-kube-api-access-4ns74" (OuterVolumeSpecName: "kube-api-access-4ns74") pod "9d8a46d6-d163-4283-89a2-a50b1d40dbe5" (UID: "9d8a46d6-d163-4283-89a2-a50b1d40dbe5"). InnerVolumeSpecName "kube-api-access-4ns74". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 08:24:25.789657 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:24:25.789625 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:24:25.866039 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:24:25.865949 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d8a46d6-d163-4283-89a2-a50b1d40dbe5-openshift-service-ca\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:24:25.866039 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:24:25.865981 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d8a46d6-d163-4283-89a2-a50b1d40dbe5-maas-api-tls\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:24:25.866039 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:24:25.865993 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ns74\" (UniqueName: \"kubernetes.io/projected/9d8a46d6-d163-4283-89a2-a50b1d40dbe5-kube-api-access-4ns74\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:24:26.559027 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:24:26.558980 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dac3f27a-2ab3-4707-84b9-ed590c895b2c" path="/var/lib/kubelet/pods/dac3f27a-2ab3-4707-84b9-ed590c895b2c/volumes" Jul 19 08:24:26.570329 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:24:26.570302 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fdbad2e2-795d7b46bb-24d7l" Jul 19 08:24:26.570497 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:24:26.570339 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fdbad2e2-795d7b46bb-24d7l" event={"ID":"9d8a46d6-d163-4283-89a2-a50b1d40dbe5","Type":"ContainerDied","Data":"655d8e4206338ae78d44f943e4f93609cf15500ec19625ad236c411e81a70731"} Jul 19 08:24:26.570497 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:24:26.570400 2574 scope.go:117] "RemoveContainer" containerID="15dad8837efcd969e12013531b57346d6d2437764273af951286e1dbcf72c9c4" Jul 19 08:24:26.586322 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:24:26.586298 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-fdbad2e2-795d7b46bb-24d7l"] Jul 19 08:24:26.591678 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:24:26.591653 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-fdbad2e2-795d7b46bb-24d7l"] Jul 19 08:24:28.558696 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:24:28.558653 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d8a46d6-d163-4283-89a2-a50b1d40dbe5" path="/var/lib/kubelet/pods/9d8a46d6-d163-4283-89a2-a50b1d40dbe5/volumes" Jul 19 08:24:31.200253 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:24:31.200213 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:24:31.267139 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:24:31.267095 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:24:31.689665 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:24:31.689606 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:24:31.789486 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:24:31.789449 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:24:31.890400 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:24:31.890360 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:24:33.259364 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:24:33.259325 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:24:36.659171 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:24:36.659133 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:24:38.820801 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:24:38.820764 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:24:38.874954 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:24:38.874908 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:24:38.954829 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:24:38.954787 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:24:39.047480 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:24:39.047432 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:24:39.148691 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:24:39.148599 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:24:39.508977 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:24:39.508943 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:25:05.435984 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:25:05.435939 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:25:11.148832 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:25:11.148795 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:25:11.206857 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:25:11.206821 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:25:11.285852 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:25:11.285801 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:25:11.387043 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:25:11.386991 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:25:27.185101 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:25:27.185058 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:25:33.034237 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:25:33.034196 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:25:33.182704 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:25:33.182665 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:25:33.284649 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:25:33.284508 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:25:43.762449 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:25:43.762358 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:25:44.545133 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:25:44.545095 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:25:50.161103 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:25:50.161061 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:25:50.261680 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:25:50.261632 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:25:50.361564 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:25:50.361501 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:25:50.461372 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:25:50.461334 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:25:51.191132 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:25:51.191078 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:25:51.280296 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:25:51.280260 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:25:51.334580 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:25:51.334545 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:25:52.901581 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:25:52.901544 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:25:54.673077 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:25:54.673039 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 08:25:58.610857 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:25:58.610823 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:25:58.711632 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:25:58.711591 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:25:59.271084 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:25:59.271045 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:26:05.105322 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:05.105279 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:26:05.206395 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:05.206355 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:26:05.331958 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:05.331915 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:26:05.609677 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:05.609635 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-6eff15b1-openshift-default-54bf58bdf5-lf927"] Jul 19 08:26:05.610107 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:05.610086 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d8a46d6-d163-4283-89a2-a50b1d40dbe5" containerName="maas-api" Jul 19 08:26:05.610203 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:05.610109 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d8a46d6-d163-4283-89a2-a50b1d40dbe5" containerName="maas-api" Jul 19 08:26:05.610203 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:05.610133 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d8a46d6-d163-4283-89a2-a50b1d40dbe5" containerName="maas-api" Jul 19 08:26:05.610203 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:05.610141 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d8a46d6-d163-4283-89a2-a50b1d40dbe5" containerName="maas-api" Jul 19 08:26:05.610203 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:05.610163 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dac3f27a-2ab3-4707-84b9-ed590c895b2c" containerName="istio-proxy" Jul 19 08:26:05.610203 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:05.610173 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="dac3f27a-2ab3-4707-84b9-ed590c895b2c" containerName="istio-proxy" Jul 19 08:26:05.610203 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:05.610188 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d8a46d6-d163-4283-89a2-a50b1d40dbe5" containerName="maas-api" Jul 19 08:26:05.610203 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:05.610196 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d8a46d6-d163-4283-89a2-a50b1d40dbe5" containerName="maas-api" Jul 19 08:26:05.610203 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:05.610205 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d8a46d6-d163-4283-89a2-a50b1d40dbe5" containerName="maas-api" Jul 19 08:26:05.610631 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:05.610214 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d8a46d6-d163-4283-89a2-a50b1d40dbe5" containerName="maas-api" Jul 19 08:26:05.610631 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:05.610308 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d8a46d6-d163-4283-89a2-a50b1d40dbe5" containerName="maas-api" Jul 19 08:26:05.610631 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:05.610319 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="dac3f27a-2ab3-4707-84b9-ed590c895b2c" containerName="istio-proxy" Jul 19 08:26:05.610631 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:05.610328 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d8a46d6-d163-4283-89a2-a50b1d40dbe5" containerName="maas-api" Jul 19 08:26:05.610631 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:05.610338 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d8a46d6-d163-4283-89a2-a50b1d40dbe5" containerName="maas-api" Jul 19 08:26:05.610631 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:05.610347 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d8a46d6-d163-4283-89a2-a50b1d40dbe5" containerName="maas-api" Jul 19 08:26:05.610631 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:05.610357 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="dac3f27a-2ab3-4707-84b9-ed590c895b2c" containerName="istio-proxy" Jul 19 08:26:05.613271 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:05.613249 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6eff15b1-openshift-default-54bf58bdf5-lf927" Jul 19 08:26:05.618346 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:05.618320 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 19 08:26:05.618486 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:05.618320 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-6eff15b1-openshift-default-dockercfg-srkgs\"" Jul 19 08:26:05.626351 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:05.626321 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6eff15b1-openshift-default-54bf58bdf5-lf927"] Jul 19 08:26:05.635862 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:05.635823 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:26:05.727882 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:05.727841 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19ae90a7-a58d-43f2-8c19-a93fca3d518f-istiod-ca-cert\") pod \"e2e-mt-6eff15b1-openshift-default-54bf58bdf5-lf927\" (UID: \"19ae90a7-a58d-43f2-8c19-a93fca3d518f\") " pod="openshift-ingress/e2e-mt-6eff15b1-openshift-default-54bf58bdf5-lf927" Jul 19 08:26:05.727882 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:05.727887 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19ae90a7-a58d-43f2-8c19-a93fca3d518f-workload-certs\") pod \"e2e-mt-6eff15b1-openshift-default-54bf58bdf5-lf927\" (UID: \"19ae90a7-a58d-43f2-8c19-a93fca3d518f\") " pod="openshift-ingress/e2e-mt-6eff15b1-openshift-default-54bf58bdf5-lf927" Jul 19 08:26:05.728125 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:05.727917 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19ae90a7-a58d-43f2-8c19-a93fca3d518f-credential-socket\") pod \"e2e-mt-6eff15b1-openshift-default-54bf58bdf5-lf927\" (UID: \"19ae90a7-a58d-43f2-8c19-a93fca3d518f\") " pod="openshift-ingress/e2e-mt-6eff15b1-openshift-default-54bf58bdf5-lf927" Jul 19 08:26:05.728125 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:05.728025 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19ae90a7-a58d-43f2-8c19-a93fca3d518f-istio-podinfo\") pod \"e2e-mt-6eff15b1-openshift-default-54bf58bdf5-lf927\" (UID: \"19ae90a7-a58d-43f2-8c19-a93fca3d518f\") " pod="openshift-ingress/e2e-mt-6eff15b1-openshift-default-54bf58bdf5-lf927" Jul 19 08:26:05.728125 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:05.728080 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmbmm\" (UniqueName: \"kubernetes.io/projected/19ae90a7-a58d-43f2-8c19-a93fca3d518f-kube-api-access-dmbmm\") pod \"e2e-mt-6eff15b1-openshift-default-54bf58bdf5-lf927\" (UID: \"19ae90a7-a58d-43f2-8c19-a93fca3d518f\") " pod="openshift-ingress/e2e-mt-6eff15b1-openshift-default-54bf58bdf5-lf927" Jul 19 08:26:05.728226 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:05.728152 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19ae90a7-a58d-43f2-8c19-a93fca3d518f-istio-token\") pod \"e2e-mt-6eff15b1-openshift-default-54bf58bdf5-lf927\" (UID: \"19ae90a7-a58d-43f2-8c19-a93fca3d518f\") " pod="openshift-ingress/e2e-mt-6eff15b1-openshift-default-54bf58bdf5-lf927" Jul 19 08:26:05.728226 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:05.728206 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19ae90a7-a58d-43f2-8c19-a93fca3d518f-workload-socket\") pod \"e2e-mt-6eff15b1-openshift-default-54bf58bdf5-lf927\" (UID: \"19ae90a7-a58d-43f2-8c19-a93fca3d518f\") " pod="openshift-ingress/e2e-mt-6eff15b1-openshift-default-54bf58bdf5-lf927" Jul 19 08:26:05.728292 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:05.728259 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19ae90a7-a58d-43f2-8c19-a93fca3d518f-istio-envoy\") pod \"e2e-mt-6eff15b1-openshift-default-54bf58bdf5-lf927\" (UID: \"19ae90a7-a58d-43f2-8c19-a93fca3d518f\") " pod="openshift-ingress/e2e-mt-6eff15b1-openshift-default-54bf58bdf5-lf927" Jul 19 08:26:05.728326 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:05.728286 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19ae90a7-a58d-43f2-8c19-a93fca3d518f-istio-data\") pod \"e2e-mt-6eff15b1-openshift-default-54bf58bdf5-lf927\" (UID: \"19ae90a7-a58d-43f2-8c19-a93fca3d518f\") " pod="openshift-ingress/e2e-mt-6eff15b1-openshift-default-54bf58bdf5-lf927" Jul 19 08:26:05.829281 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:05.829244 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19ae90a7-a58d-43f2-8c19-a93fca3d518f-workload-socket\") pod \"e2e-mt-6eff15b1-openshift-default-54bf58bdf5-lf927\" (UID: \"19ae90a7-a58d-43f2-8c19-a93fca3d518f\") " pod="openshift-ingress/e2e-mt-6eff15b1-openshift-default-54bf58bdf5-lf927" Jul 19 08:26:05.829281 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:05.829285 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19ae90a7-a58d-43f2-8c19-a93fca3d518f-istio-envoy\") pod \"e2e-mt-6eff15b1-openshift-default-54bf58bdf5-lf927\" (UID: \"19ae90a7-a58d-43f2-8c19-a93fca3d518f\") " pod="openshift-ingress/e2e-mt-6eff15b1-openshift-default-54bf58bdf5-lf927" Jul 19 08:26:05.829556 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:05.829304 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19ae90a7-a58d-43f2-8c19-a93fca3d518f-istio-data\") pod \"e2e-mt-6eff15b1-openshift-default-54bf58bdf5-lf927\" (UID: \"19ae90a7-a58d-43f2-8c19-a93fca3d518f\") " pod="openshift-ingress/e2e-mt-6eff15b1-openshift-default-54bf58bdf5-lf927" Jul 19 08:26:05.829556 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:05.829328 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19ae90a7-a58d-43f2-8c19-a93fca3d518f-istiod-ca-cert\") pod \"e2e-mt-6eff15b1-openshift-default-54bf58bdf5-lf927\" (UID: \"19ae90a7-a58d-43f2-8c19-a93fca3d518f\") " pod="openshift-ingress/e2e-mt-6eff15b1-openshift-default-54bf58bdf5-lf927" Jul 19 08:26:05.829556 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:05.829355 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19ae90a7-a58d-43f2-8c19-a93fca3d518f-workload-certs\") pod \"e2e-mt-6eff15b1-openshift-default-54bf58bdf5-lf927\" (UID: \"19ae90a7-a58d-43f2-8c19-a93fca3d518f\") " pod="openshift-ingress/e2e-mt-6eff15b1-openshift-default-54bf58bdf5-lf927" Jul 19 08:26:05.829556 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:05.829393 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19ae90a7-a58d-43f2-8c19-a93fca3d518f-credential-socket\") pod \"e2e-mt-6eff15b1-openshift-default-54bf58bdf5-lf927\" (UID: \"19ae90a7-a58d-43f2-8c19-a93fca3d518f\") " pod="openshift-ingress/e2e-mt-6eff15b1-openshift-default-54bf58bdf5-lf927" Jul 19 08:26:05.829556 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:05.829430 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19ae90a7-a58d-43f2-8c19-a93fca3d518f-istio-podinfo\") pod \"e2e-mt-6eff15b1-openshift-default-54bf58bdf5-lf927\" (UID: \"19ae90a7-a58d-43f2-8c19-a93fca3d518f\") " pod="openshift-ingress/e2e-mt-6eff15b1-openshift-default-54bf58bdf5-lf927" Jul 19 08:26:05.829556 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:05.829456 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmbmm\" (UniqueName: \"kubernetes.io/projected/19ae90a7-a58d-43f2-8c19-a93fca3d518f-kube-api-access-dmbmm\") pod \"e2e-mt-6eff15b1-openshift-default-54bf58bdf5-lf927\" (UID: \"19ae90a7-a58d-43f2-8c19-a93fca3d518f\") " pod="openshift-ingress/e2e-mt-6eff15b1-openshift-default-54bf58bdf5-lf927" Jul 19 08:26:05.829556 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:05.829533 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19ae90a7-a58d-43f2-8c19-a93fca3d518f-istio-token\") pod \"e2e-mt-6eff15b1-openshift-default-54bf58bdf5-lf927\" (UID: \"19ae90a7-a58d-43f2-8c19-a93fca3d518f\") " pod="openshift-ingress/e2e-mt-6eff15b1-openshift-default-54bf58bdf5-lf927" Jul 19 08:26:05.829942 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:05.829858 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19ae90a7-a58d-43f2-8c19-a93fca3d518f-istio-data\") pod \"e2e-mt-6eff15b1-openshift-default-54bf58bdf5-lf927\" (UID: \"19ae90a7-a58d-43f2-8c19-a93fca3d518f\") " pod="openshift-ingress/e2e-mt-6eff15b1-openshift-default-54bf58bdf5-lf927" Jul 19 08:26:05.829942 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:05.829882 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19ae90a7-a58d-43f2-8c19-a93fca3d518f-workload-socket\") pod \"e2e-mt-6eff15b1-openshift-default-54bf58bdf5-lf927\" (UID: \"19ae90a7-a58d-43f2-8c19-a93fca3d518f\") " pod="openshift-ingress/e2e-mt-6eff15b1-openshift-default-54bf58bdf5-lf927" Jul 19 08:26:05.830145 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:05.830121 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19ae90a7-a58d-43f2-8c19-a93fca3d518f-credential-socket\") pod \"e2e-mt-6eff15b1-openshift-default-54bf58bdf5-lf927\" (UID: \"19ae90a7-a58d-43f2-8c19-a93fca3d518f\") " pod="openshift-ingress/e2e-mt-6eff15b1-openshift-default-54bf58bdf5-lf927" Jul 19 08:26:05.830245 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:05.830160 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19ae90a7-a58d-43f2-8c19-a93fca3d518f-workload-certs\") pod \"e2e-mt-6eff15b1-openshift-default-54bf58bdf5-lf927\" (UID: \"19ae90a7-a58d-43f2-8c19-a93fca3d518f\") " pod="openshift-ingress/e2e-mt-6eff15b1-openshift-default-54bf58bdf5-lf927" Jul 19 08:26:05.830737 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:05.830713 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19ae90a7-a58d-43f2-8c19-a93fca3d518f-istiod-ca-cert\") pod \"e2e-mt-6eff15b1-openshift-default-54bf58bdf5-lf927\" (UID: \"19ae90a7-a58d-43f2-8c19-a93fca3d518f\") " pod="openshift-ingress/e2e-mt-6eff15b1-openshift-default-54bf58bdf5-lf927" Jul 19 08:26:05.831891 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:05.831868 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19ae90a7-a58d-43f2-8c19-a93fca3d518f-istio-envoy\") pod \"e2e-mt-6eff15b1-openshift-default-54bf58bdf5-lf927\" (UID: \"19ae90a7-a58d-43f2-8c19-a93fca3d518f\") " pod="openshift-ingress/e2e-mt-6eff15b1-openshift-default-54bf58bdf5-lf927" Jul 19 08:26:05.832820 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:05.832795 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19ae90a7-a58d-43f2-8c19-a93fca3d518f-istio-podinfo\") pod \"e2e-mt-6eff15b1-openshift-default-54bf58bdf5-lf927\" (UID: \"19ae90a7-a58d-43f2-8c19-a93fca3d518f\") " pod="openshift-ingress/e2e-mt-6eff15b1-openshift-default-54bf58bdf5-lf927" Jul 19 08:26:05.837556 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:05.837499 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmbmm\" (UniqueName: \"kubernetes.io/projected/19ae90a7-a58d-43f2-8c19-a93fca3d518f-kube-api-access-dmbmm\") pod \"e2e-mt-6eff15b1-openshift-default-54bf58bdf5-lf927\" (UID: \"19ae90a7-a58d-43f2-8c19-a93fca3d518f\") " pod="openshift-ingress/e2e-mt-6eff15b1-openshift-default-54bf58bdf5-lf927" Jul 19 08:26:05.837642 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:05.837603 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19ae90a7-a58d-43f2-8c19-a93fca3d518f-istio-token\") pod \"e2e-mt-6eff15b1-openshift-default-54bf58bdf5-lf927\" (UID: \"19ae90a7-a58d-43f2-8c19-a93fca3d518f\") " pod="openshift-ingress/e2e-mt-6eff15b1-openshift-default-54bf58bdf5-lf927" Jul 19 08:26:05.911479 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:05.911382 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:26:05.927284 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:05.927249 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6eff15b1-openshift-default-54bf58bdf5-lf927" Jul 19 08:26:06.070005 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:06.069979 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6eff15b1-openshift-default-54bf58bdf5-lf927"] Jul 19 08:26:06.070330 ip-10-0-136-70 kubenswrapper[2574]: W0719 08:26:06.070301 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19ae90a7_a58d_43f2_8c19_a93fca3d518f.slice/crio-d702492f9c2accdf8622e2e19964fe8275e5f30d145162ffe1e9e989214c8834 WatchSource:0}: Error finding container d702492f9c2accdf8622e2e19964fe8275e5f30d145162ffe1e9e989214c8834: Status 404 returned error can't find the container with id d702492f9c2accdf8622e2e19964fe8275e5f30d145162ffe1e9e989214c8834 Jul 19 08:26:06.072326 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:06.072290 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 19 08:26:06.072436 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:06.072355 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 19 08:26:06.072436 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:06.072385 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 19 08:26:07.072623 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:07.072577 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6eff15b1-openshift-default-54bf58bdf5-lf927" event={"ID":"19ae90a7-a58d-43f2-8c19-a93fca3d518f","Type":"ContainerStarted","Data":"c1ef44dcf181f0deda0c1a5ecce70ff907e0c54580f488bb41575ec013a43801"} Jul 19 08:26:07.073088 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:07.072639 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6eff15b1-openshift-default-54bf58bdf5-lf927" event={"ID":"19ae90a7-a58d-43f2-8c19-a93fca3d518f","Type":"ContainerStarted","Data":"d702492f9c2accdf8622e2e19964fe8275e5f30d145162ffe1e9e989214c8834"} Jul 19 08:26:07.094175 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:07.094120 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-6eff15b1-openshift-default-54bf58bdf5-lf927" podStartSLOduration=2.094103854 podStartE2EDuration="2.094103854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 08:26:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:26:07.092572753 +0000 UTC m=+2633.446882361" watchObservedRunningTime="2026-07-19 08:26:07.094103854 +0000 UTC m=+2633.448413454" Jul 19 08:26:07.863756 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:07.863715 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 08:26:07.927958 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:07.927906 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-6eff15b1-openshift-default-54bf58bdf5-lf927" Jul 19 08:26:07.933770 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:07.933742 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-6eff15b1-openshift-default-54bf58bdf5-lf927" Jul 19 08:26:08.077995 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:08.077958 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-6eff15b1-openshift-default-54bf58bdf5-lf927" Jul 19 08:26:08.079009 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:08.078988 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-6eff15b1-openshift-default-54bf58bdf5-lf927" Jul 19 08:26:08.981246 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:08.981199 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:26:09.070589 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:09.070547 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:26:09.072918 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:09.072895 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6eff15b1-6bd8bfbd5b-xl8ns"] Jul 19 08:26:09.073471 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:09.073452 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dac3f27a-2ab3-4707-84b9-ed590c895b2c" containerName="istio-proxy" Jul 19 08:26:09.073526 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:09.073476 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="dac3f27a-2ab3-4707-84b9-ed590c895b2c" containerName="istio-proxy" Jul 19 08:26:09.073577 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:09.073546 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d8a46d6-d163-4283-89a2-a50b1d40dbe5" containerName="maas-api" Jul 19 08:26:09.073577 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:09.073556 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d8a46d6-d163-4283-89a2-a50b1d40dbe5" containerName="maas-api" Jul 19 08:26:09.073678 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:09.073667 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d8a46d6-d163-4283-89a2-a50b1d40dbe5" containerName="maas-api" Jul 19 08:26:09.077434 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:09.077409 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6eff15b1-6bd8bfbd5b-xl8ns" Jul 19 08:26:09.080206 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:09.080182 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 19 08:26:09.081179 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:09.081147 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-6eff15b1-serving-cert\"" Jul 19 08:26:09.081319 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:09.081190 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 19 08:26:09.081319 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:09.081155 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-tdbkh\"" Jul 19 08:26:09.089511 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:09.089190 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6eff15b1-6bd8bfbd5b-xl8ns"] Jul 19 08:26:09.145182 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:09.145139 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:26:09.159917 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:09.159883 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/374d5762-6a34-4049-93f2-924a49b8e40a-openshift-service-ca\") pod \"maas-api-e2e-mt-6eff15b1-6bd8bfbd5b-xl8ns\" (UID: \"374d5762-6a34-4049-93f2-924a49b8e40a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6eff15b1-6bd8bfbd5b-xl8ns" Jul 19 08:26:09.160095 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:09.160066 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/374d5762-6a34-4049-93f2-924a49b8e40a-maas-api-tls\") pod \"maas-api-e2e-mt-6eff15b1-6bd8bfbd5b-xl8ns\" (UID: \"374d5762-6a34-4049-93f2-924a49b8e40a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6eff15b1-6bd8bfbd5b-xl8ns" Jul 19 08:26:09.160150 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:09.160112 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wnb6\" (UniqueName: \"kubernetes.io/projected/374d5762-6a34-4049-93f2-924a49b8e40a-kube-api-access-7wnb6\") pod \"maas-api-e2e-mt-6eff15b1-6bd8bfbd5b-xl8ns\" (UID: \"374d5762-6a34-4049-93f2-924a49b8e40a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6eff15b1-6bd8bfbd5b-xl8ns" Jul 19 08:26:09.260699 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:09.260614 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/374d5762-6a34-4049-93f2-924a49b8e40a-maas-api-tls\") pod \"maas-api-e2e-mt-6eff15b1-6bd8bfbd5b-xl8ns\" (UID: \"374d5762-6a34-4049-93f2-924a49b8e40a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6eff15b1-6bd8bfbd5b-xl8ns" Jul 19 08:26:09.260699 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:09.260658 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wnb6\" (UniqueName: \"kubernetes.io/projected/374d5762-6a34-4049-93f2-924a49b8e40a-kube-api-access-7wnb6\") pod \"maas-api-e2e-mt-6eff15b1-6bd8bfbd5b-xl8ns\" (UID: \"374d5762-6a34-4049-93f2-924a49b8e40a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6eff15b1-6bd8bfbd5b-xl8ns" Jul 19 08:26:09.260918 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:09.260751 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/374d5762-6a34-4049-93f2-924a49b8e40a-openshift-service-ca\") pod \"maas-api-e2e-mt-6eff15b1-6bd8bfbd5b-xl8ns\" (UID: \"374d5762-6a34-4049-93f2-924a49b8e40a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6eff15b1-6bd8bfbd5b-xl8ns" Jul 19 08:26:09.260918 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:26:09.260763 2574 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-6eff15b1-serving-cert: secret "maas-api-e2e-mt-6eff15b1-serving-cert" not found Jul 19 08:26:09.260918 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:26:09.260842 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/374d5762-6a34-4049-93f2-924a49b8e40a-maas-api-tls podName:374d5762-6a34-4049-93f2-924a49b8e40a nodeName:}" failed. No retries permitted until 2026-07-19 08:26:09.760820599 +0000 UTC m=+2636.115130178 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/374d5762-6a34-4049-93f2-924a49b8e40a-maas-api-tls") pod "maas-api-e2e-mt-6eff15b1-6bd8bfbd5b-xl8ns" (UID: "374d5762-6a34-4049-93f2-924a49b8e40a") : secret "maas-api-e2e-mt-6eff15b1-serving-cert" not found Jul 19 08:26:09.261431 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:09.261411 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/374d5762-6a34-4049-93f2-924a49b8e40a-openshift-service-ca\") pod \"maas-api-e2e-mt-6eff15b1-6bd8bfbd5b-xl8ns\" (UID: \"374d5762-6a34-4049-93f2-924a49b8e40a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6eff15b1-6bd8bfbd5b-xl8ns" Jul 19 08:26:09.269857 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:09.269829 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wnb6\" (UniqueName: \"kubernetes.io/projected/374d5762-6a34-4049-93f2-924a49b8e40a-kube-api-access-7wnb6\") pod \"maas-api-e2e-mt-6eff15b1-6bd8bfbd5b-xl8ns\" (UID: \"374d5762-6a34-4049-93f2-924a49b8e40a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6eff15b1-6bd8bfbd5b-xl8ns" Jul 19 08:26:09.361244 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:09.361208 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:26:09.645856 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:09.645760 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:26:09.747102 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:09.747054 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:26:09.765127 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:09.765093 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/374d5762-6a34-4049-93f2-924a49b8e40a-maas-api-tls\") pod \"maas-api-e2e-mt-6eff15b1-6bd8bfbd5b-xl8ns\" (UID: \"374d5762-6a34-4049-93f2-924a49b8e40a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6eff15b1-6bd8bfbd5b-xl8ns" Jul 19 08:26:09.767589 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:09.767555 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/374d5762-6a34-4049-93f2-924a49b8e40a-maas-api-tls\") pod \"maas-api-e2e-mt-6eff15b1-6bd8bfbd5b-xl8ns\" (UID: \"374d5762-6a34-4049-93f2-924a49b8e40a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6eff15b1-6bd8bfbd5b-xl8ns" Jul 19 08:26:09.991935 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:09.991897 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6eff15b1-6bd8bfbd5b-xl8ns" Jul 19 08:26:10.154762 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:10.154738 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6eff15b1-6bd8bfbd5b-xl8ns"] Jul 19 08:26:10.156757 ip-10-0-136-70 kubenswrapper[2574]: W0719 08:26:10.156724 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod374d5762_6a34_4049_93f2_924a49b8e40a.slice/crio-7b8c3a42bd7e64819381eefeda4e756dac5f8499ccf235a8abfaaf243ad1fb08 WatchSource:0}: Error finding container 7b8c3a42bd7e64819381eefeda4e756dac5f8499ccf235a8abfaaf243ad1fb08: Status 404 returned error can't find the container with id 7b8c3a42bd7e64819381eefeda4e756dac5f8499ccf235a8abfaaf243ad1fb08 Jul 19 08:26:10.158186 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:10.158169 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 19 08:26:11.097176 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:11.097137 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6eff15b1-6bd8bfbd5b-xl8ns" event={"ID":"374d5762-6a34-4049-93f2-924a49b8e40a","Type":"ContainerStarted","Data":"df0658e4790e644cb0d0513ba7b69241bb8dceff250e28a2a36155cc40ce1ea4"} Jul 19 08:26:11.097368 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:11.097202 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6eff15b1-6bd8bfbd5b-xl8ns" event={"ID":"374d5762-6a34-4049-93f2-924a49b8e40a","Type":"ContainerStarted","Data":"7b8c3a42bd7e64819381eefeda4e756dac5f8499ccf235a8abfaaf243ad1fb08"} Jul 19 08:26:11.097368 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:11.097336 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6eff15b1-6bd8bfbd5b-xl8ns" Jul 19 08:26:11.114683 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:11.114631 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6eff15b1-6bd8bfbd5b-xl8ns" podStartSLOduration=1.651915735 podStartE2EDuration="2.114615993s" totalImagesPullingTime="462.700258ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 08:26:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:26:11.112490788 +0000 UTC m=+2637.466800389" watchObservedRunningTime="2026-07-19 08:26:11.114615993 +0000 UTC m=+2637.468925620" Jul 19 08:26:13.437241 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:13.437165 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:26:14.053332 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:14.053290 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:26:14.152984 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:14.152945 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:26:14.459565 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:14.459453 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:26:17.113949 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:17.113921 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6eff15b1-6bd8bfbd5b-xl8ns" Jul 19 08:26:19.870655 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:19.870613 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 08:26:20.156991 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:20.156880 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:26:20.257251 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:20.257215 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:26:22.171368 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:22.171270 2574 generic.go:356] "Generic (PLEG): container finished" podID="19ae90a7-a58d-43f2-8c19-a93fca3d518f" containerID="c1ef44dcf181f0deda0c1a5ecce70ff907e0c54580f488bb41575ec013a43801" exitCode=0 Jul 19 08:26:22.171368 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:22.171325 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6eff15b1-openshift-default-54bf58bdf5-lf927" event={"ID":"19ae90a7-a58d-43f2-8c19-a93fca3d518f","Type":"ContainerDied","Data":"c1ef44dcf181f0deda0c1a5ecce70ff907e0c54580f488bb41575ec013a43801"} Jul 19 08:26:22.171839 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:22.171722 2574 scope.go:117] "RemoveContainer" containerID="c1ef44dcf181f0deda0c1a5ecce70ff907e0c54580f488bb41575ec013a43801" Jul 19 08:26:22.172785 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:22.172754 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 19 08:26:22.172862 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:22.172831 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 19 08:26:22.172902 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:22.172864 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 19 08:26:23.078746 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:23.078693 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-6eff15b1-openshift-default-54bf58bdf5-lf927" Jul 19 08:26:23.178724 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:23.178686 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6eff15b1-openshift-default-54bf58bdf5-lf927" event={"ID":"19ae90a7-a58d-43f2-8c19-a93fca3d518f","Type":"ContainerStarted","Data":"f0ad37b0cf696ebb1d4db2d24127328e1258041888397427b9908d1a3e09efba"} Jul 19 08:26:23.927827 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:23.927706 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-6eff15b1-openshift-default-54bf58bdf5-lf927" Jul 19 08:26:23.929457 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:23.929036 2574 patch_prober.go:28] interesting pod/e2e-mt-6eff15b1-openshift-default-54bf58bdf5-lf927 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Jul 19 08:26:23.929457 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:23.929115 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6eff15b1-openshift-default-54bf58bdf5-lf927" podUID="19ae90a7-a58d-43f2-8c19-a93fca3d518f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Jul 19 08:26:24.928702 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:24.928659 2574 patch_prober.go:28] interesting pod/e2e-mt-6eff15b1-openshift-default-54bf58bdf5-lf927 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Jul 19 08:26:24.929344 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:24.929304 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6eff15b1-openshift-default-54bf58bdf5-lf927" podUID="19ae90a7-a58d-43f2-8c19-a93fca3d518f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Jul 19 08:26:25.927933 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:25.927889 2574 patch_prober.go:28] interesting pod/e2e-mt-6eff15b1-openshift-default-54bf58bdf5-lf927 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Jul 19 08:26:25.928174 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:25.927959 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6eff15b1-openshift-default-54bf58bdf5-lf927" podUID="19ae90a7-a58d-43f2-8c19-a93fca3d518f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Jul 19 08:26:26.928725 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:26.928679 2574 patch_prober.go:28] interesting pod/e2e-mt-6eff15b1-openshift-default-54bf58bdf5-lf927 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Jul 19 08:26:26.929260 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:26.928764 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6eff15b1-openshift-default-54bf58bdf5-lf927" podUID="19ae90a7-a58d-43f2-8c19-a93fca3d518f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Jul 19 08:26:27.931710 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:27.931677 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-6eff15b1-openshift-default-54bf58bdf5-lf927" Jul 19 08:26:27.932168 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:27.931957 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-6eff15b1-openshift-default-54bf58bdf5-lf927" Jul 19 08:26:27.932799 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:27.932777 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-6eff15b1-openshift-default-54bf58bdf5-lf927" Jul 19 08:26:31.054041 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:31.053989 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6eff15b1-6bd8bfbd5b-xl8ns"] Jul 19 08:26:31.054543 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:31.054299 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6eff15b1-6bd8bfbd5b-xl8ns" podUID="374d5762-6a34-4049-93f2-924a49b8e40a" containerName="maas-api" containerID="cri-o://df0658e4790e644cb0d0513ba7b69241bb8dceff250e28a2a36155cc40ce1ea4" gracePeriod=30 Jul 19 08:26:31.084498 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:31.084463 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:26:31.219915 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:31.219874 2574 generic.go:356] "Generic (PLEG): container finished" podID="374d5762-6a34-4049-93f2-924a49b8e40a" containerID="df0658e4790e644cb0d0513ba7b69241bb8dceff250e28a2a36155cc40ce1ea4" exitCode=0 Jul 19 08:26:31.220118 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:31.219958 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6eff15b1-6bd8bfbd5b-xl8ns" event={"ID":"374d5762-6a34-4049-93f2-924a49b8e40a","Type":"ContainerDied","Data":"df0658e4790e644cb0d0513ba7b69241bb8dceff250e28a2a36155cc40ce1ea4"} Jul 19 08:26:31.313786 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:31.313717 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6eff15b1-6bd8bfbd5b-xl8ns" Jul 19 08:26:31.374024 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:31.373990 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7wnb6\" (UniqueName: \"kubernetes.io/projected/374d5762-6a34-4049-93f2-924a49b8e40a-kube-api-access-7wnb6\") pod \"374d5762-6a34-4049-93f2-924a49b8e40a\" (UID: \"374d5762-6a34-4049-93f2-924a49b8e40a\") " Jul 19 08:26:31.374196 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:31.374037 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/374d5762-6a34-4049-93f2-924a49b8e40a-maas-api-tls\") pod \"374d5762-6a34-4049-93f2-924a49b8e40a\" (UID: \"374d5762-6a34-4049-93f2-924a49b8e40a\") " Jul 19 08:26:31.374196 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:31.374068 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/374d5762-6a34-4049-93f2-924a49b8e40a-openshift-service-ca\") pod \"374d5762-6a34-4049-93f2-924a49b8e40a\" (UID: \"374d5762-6a34-4049-93f2-924a49b8e40a\") " Jul 19 08:26:31.374534 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:31.374489 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/374d5762-6a34-4049-93f2-924a49b8e40a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "374d5762-6a34-4049-93f2-924a49b8e40a" (UID: "374d5762-6a34-4049-93f2-924a49b8e40a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 08:26:31.376285 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:31.376261 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/374d5762-6a34-4049-93f2-924a49b8e40a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "374d5762-6a34-4049-93f2-924a49b8e40a" (UID: "374d5762-6a34-4049-93f2-924a49b8e40a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 08:26:31.376285 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:31.376274 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/374d5762-6a34-4049-93f2-924a49b8e40a-kube-api-access-7wnb6" (OuterVolumeSpecName: "kube-api-access-7wnb6") pod "374d5762-6a34-4049-93f2-924a49b8e40a" (UID: "374d5762-6a34-4049-93f2-924a49b8e40a"). InnerVolumeSpecName "kube-api-access-7wnb6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 08:26:31.475066 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:31.475020 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wnb6\" (UniqueName: \"kubernetes.io/projected/374d5762-6a34-4049-93f2-924a49b8e40a-kube-api-access-7wnb6\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:26:31.475066 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:31.475060 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/374d5762-6a34-4049-93f2-924a49b8e40a-maas-api-tls\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:26:31.475066 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:31.475070 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/374d5762-6a34-4049-93f2-924a49b8e40a-openshift-service-ca\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:26:32.226142 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:32.226107 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6eff15b1-6bd8bfbd5b-xl8ns" Jul 19 08:26:32.226593 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:32.226146 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6eff15b1-6bd8bfbd5b-xl8ns" event={"ID":"374d5762-6a34-4049-93f2-924a49b8e40a","Type":"ContainerDied","Data":"7b8c3a42bd7e64819381eefeda4e756dac5f8499ccf235a8abfaaf243ad1fb08"} Jul 19 08:26:32.226593 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:32.226214 2574 scope.go:117] "RemoveContainer" containerID="df0658e4790e644cb0d0513ba7b69241bb8dceff250e28a2a36155cc40ce1ea4" Jul 19 08:26:32.248029 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:32.248000 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6eff15b1-6bd8bfbd5b-xl8ns"] Jul 19 08:26:32.258074 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:32.258048 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6eff15b1-6bd8bfbd5b-xl8ns"] Jul 19 08:26:32.559284 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:32.559197 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="374d5762-6a34-4049-93f2-924a49b8e40a" path="/var/lib/kubelet/pods/374d5762-6a34-4049-93f2-924a49b8e40a/volumes" Jul 19 08:26:34.278421 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:34.278384 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 08:26:35.577400 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:35.577366 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:26:41.292880 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:41.292841 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:26:41.356982 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:41.356942 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:26:46.223397 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:46.223359 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6eff15b1-openshift-default-54bf58bdf5-lf927"] Jul 19 08:26:46.223943 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:46.223678 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-6eff15b1-openshift-default-54bf58bdf5-lf927" podUID="19ae90a7-a58d-43f2-8c19-a93fca3d518f" containerName="istio-proxy" containerID="cri-o://f0ad37b0cf696ebb1d4db2d24127328e1258041888397427b9908d1a3e09efba" gracePeriod=30 Jul 19 08:26:46.242995 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:46.242955 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:26:46.467763 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:46.467731 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:26:46.589167 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:46.589054 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:26:46.680920 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:46.680882 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:26:46.786913 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:46.786879 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:26:51.484936 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:51.484866 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6eff15b1-openshift-default-54bf58bdf5-lf927" Jul 19 08:26:51.537283 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:51.537248 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19ae90a7-a58d-43f2-8c19-a93fca3d518f-istio-podinfo\") pod \"19ae90a7-a58d-43f2-8c19-a93fca3d518f\" (UID: \"19ae90a7-a58d-43f2-8c19-a93fca3d518f\") " Jul 19 08:26:51.537461 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:51.537313 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19ae90a7-a58d-43f2-8c19-a93fca3d518f-credential-socket\") pod \"19ae90a7-a58d-43f2-8c19-a93fca3d518f\" (UID: \"19ae90a7-a58d-43f2-8c19-a93fca3d518f\") " Jul 19 08:26:51.537461 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:51.537345 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19ae90a7-a58d-43f2-8c19-a93fca3d518f-istiod-ca-cert\") pod \"19ae90a7-a58d-43f2-8c19-a93fca3d518f\" (UID: \"19ae90a7-a58d-43f2-8c19-a93fca3d518f\") " Jul 19 08:26:51.537461 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:51.537366 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dmbmm\" (UniqueName: \"kubernetes.io/projected/19ae90a7-a58d-43f2-8c19-a93fca3d518f-kube-api-access-dmbmm\") pod \"19ae90a7-a58d-43f2-8c19-a93fca3d518f\" (UID: \"19ae90a7-a58d-43f2-8c19-a93fca3d518f\") " Jul 19 08:26:51.537461 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:51.537397 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19ae90a7-a58d-43f2-8c19-a93fca3d518f-istio-envoy\") pod \"19ae90a7-a58d-43f2-8c19-a93fca3d518f\" (UID: \"19ae90a7-a58d-43f2-8c19-a93fca3d518f\") " Jul 19 08:26:51.537461 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:51.537424 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19ae90a7-a58d-43f2-8c19-a93fca3d518f-workload-certs\") pod \"19ae90a7-a58d-43f2-8c19-a93fca3d518f\" (UID: \"19ae90a7-a58d-43f2-8c19-a93fca3d518f\") " Jul 19 08:26:51.537461 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:51.537458 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19ae90a7-a58d-43f2-8c19-a93fca3d518f-istio-data\") pod \"19ae90a7-a58d-43f2-8c19-a93fca3d518f\" (UID: \"19ae90a7-a58d-43f2-8c19-a93fca3d518f\") " Jul 19 08:26:51.537776 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:51.537514 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19ae90a7-a58d-43f2-8c19-a93fca3d518f-workload-socket\") pod \"19ae90a7-a58d-43f2-8c19-a93fca3d518f\" (UID: \"19ae90a7-a58d-43f2-8c19-a93fca3d518f\") " Jul 19 08:26:51.537776 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:51.537583 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19ae90a7-a58d-43f2-8c19-a93fca3d518f-istio-token\") pod \"19ae90a7-a58d-43f2-8c19-a93fca3d518f\" (UID: \"19ae90a7-a58d-43f2-8c19-a93fca3d518f\") " Jul 19 08:26:51.537776 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:51.537706 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19ae90a7-a58d-43f2-8c19-a93fca3d518f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "19ae90a7-a58d-43f2-8c19-a93fca3d518f" (UID: "19ae90a7-a58d-43f2-8c19-a93fca3d518f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 08:26:51.537776 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:51.537727 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19ae90a7-a58d-43f2-8c19-a93fca3d518f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "19ae90a7-a58d-43f2-8c19-a93fca3d518f" (UID: "19ae90a7-a58d-43f2-8c19-a93fca3d518f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 08:26:51.537973 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:51.537831 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19ae90a7-a58d-43f2-8c19-a93fca3d518f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "19ae90a7-a58d-43f2-8c19-a93fca3d518f" (UID: "19ae90a7-a58d-43f2-8c19-a93fca3d518f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 08:26:51.537973 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:51.537917 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19ae90a7-a58d-43f2-8c19-a93fca3d518f-workload-certs\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:26:51.537973 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:51.537939 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19ae90a7-a58d-43f2-8c19-a93fca3d518f-credential-socket\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:26:51.537973 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:51.537956 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19ae90a7-a58d-43f2-8c19-a93fca3d518f-istiod-ca-cert\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:26:51.537973 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:51.537940 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19ae90a7-a58d-43f2-8c19-a93fca3d518f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "19ae90a7-a58d-43f2-8c19-a93fca3d518f" (UID: "19ae90a7-a58d-43f2-8c19-a93fca3d518f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 08:26:51.538606 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:51.538572 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19ae90a7-a58d-43f2-8c19-a93fca3d518f-istio-data" (OuterVolumeSpecName: "istio-data") pod "19ae90a7-a58d-43f2-8c19-a93fca3d518f" (UID: "19ae90a7-a58d-43f2-8c19-a93fca3d518f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 08:26:51.539768 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:51.539740 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19ae90a7-a58d-43f2-8c19-a93fca3d518f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "19ae90a7-a58d-43f2-8c19-a93fca3d518f" (UID: "19ae90a7-a58d-43f2-8c19-a93fca3d518f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 08:26:51.539870 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:51.539765 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19ae90a7-a58d-43f2-8c19-a93fca3d518f-kube-api-access-dmbmm" (OuterVolumeSpecName: "kube-api-access-dmbmm") pod "19ae90a7-a58d-43f2-8c19-a93fca3d518f" (UID: "19ae90a7-a58d-43f2-8c19-a93fca3d518f"). InnerVolumeSpecName "kube-api-access-dmbmm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 08:26:51.539870 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:51.539797 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/19ae90a7-a58d-43f2-8c19-a93fca3d518f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "19ae90a7-a58d-43f2-8c19-a93fca3d518f" (UID: "19ae90a7-a58d-43f2-8c19-a93fca3d518f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 19 08:26:51.539870 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:51.539797 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19ae90a7-a58d-43f2-8c19-a93fca3d518f-istio-token" (OuterVolumeSpecName: "istio-token") pod "19ae90a7-a58d-43f2-8c19-a93fca3d518f" (UID: "19ae90a7-a58d-43f2-8c19-a93fca3d518f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 08:26:51.638395 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:51.638364 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19ae90a7-a58d-43f2-8c19-a93fca3d518f-istio-data\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:26:51.638395 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:51.638395 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19ae90a7-a58d-43f2-8c19-a93fca3d518f-workload-socket\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:26:51.638648 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:51.638406 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19ae90a7-a58d-43f2-8c19-a93fca3d518f-istio-token\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:26:51.638648 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:51.638419 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19ae90a7-a58d-43f2-8c19-a93fca3d518f-istio-podinfo\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:26:51.638648 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:51.638430 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmbmm\" (UniqueName: \"kubernetes.io/projected/19ae90a7-a58d-43f2-8c19-a93fca3d518f-kube-api-access-dmbmm\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:26:51.638648 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:51.638441 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19ae90a7-a58d-43f2-8c19-a93fca3d518f-istio-envoy\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:26:52.326132 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:52.326095 2574 generic.go:356] "Generic (PLEG): container finished" podID="19ae90a7-a58d-43f2-8c19-a93fca3d518f" containerID="f0ad37b0cf696ebb1d4db2d24127328e1258041888397427b9908d1a3e09efba" exitCode=0 Jul 19 08:26:52.326132 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:52.326141 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6eff15b1-openshift-default-54bf58bdf5-lf927" Jul 19 08:26:52.326367 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:52.326175 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6eff15b1-openshift-default-54bf58bdf5-lf927" event={"ID":"19ae90a7-a58d-43f2-8c19-a93fca3d518f","Type":"ContainerDied","Data":"f0ad37b0cf696ebb1d4db2d24127328e1258041888397427b9908d1a3e09efba"} Jul 19 08:26:52.326367 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:52.326235 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6eff15b1-openshift-default-54bf58bdf5-lf927" event={"ID":"19ae90a7-a58d-43f2-8c19-a93fca3d518f","Type":"ContainerDied","Data":"d702492f9c2accdf8622e2e19964fe8275e5f30d145162ffe1e9e989214c8834"} Jul 19 08:26:52.326367 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:52.326264 2574 scope.go:117] "RemoveContainer" containerID="f0ad37b0cf696ebb1d4db2d24127328e1258041888397427b9908d1a3e09efba" Jul 19 08:26:52.335911 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:52.335895 2574 scope.go:117] "RemoveContainer" containerID="c1ef44dcf181f0deda0c1a5ecce70ff907e0c54580f488bb41575ec013a43801" Jul 19 08:26:52.344408 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:52.344206 2574 scope.go:117] "RemoveContainer" containerID="f0ad37b0cf696ebb1d4db2d24127328e1258041888397427b9908d1a3e09efba" Jul 19 08:26:52.344491 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:26:52.344474 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0ad37b0cf696ebb1d4db2d24127328e1258041888397427b9908d1a3e09efba\": container with ID starting with f0ad37b0cf696ebb1d4db2d24127328e1258041888397427b9908d1a3e09efba not found: ID does not exist" containerID="f0ad37b0cf696ebb1d4db2d24127328e1258041888397427b9908d1a3e09efba" Jul 19 08:26:52.344546 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:52.344499 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0ad37b0cf696ebb1d4db2d24127328e1258041888397427b9908d1a3e09efba"} err="failed to get container status \"f0ad37b0cf696ebb1d4db2d24127328e1258041888397427b9908d1a3e09efba\": rpc error: code = NotFound desc = could not find container \"f0ad37b0cf696ebb1d4db2d24127328e1258041888397427b9908d1a3e09efba\": container with ID starting with f0ad37b0cf696ebb1d4db2d24127328e1258041888397427b9908d1a3e09efba not found: ID does not exist" Jul 19 08:26:52.344546 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:52.344514 2574 scope.go:117] "RemoveContainer" containerID="c1ef44dcf181f0deda0c1a5ecce70ff907e0c54580f488bb41575ec013a43801" Jul 19 08:26:52.344755 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:26:52.344736 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1ef44dcf181f0deda0c1a5ecce70ff907e0c54580f488bb41575ec013a43801\": container with ID starting with c1ef44dcf181f0deda0c1a5ecce70ff907e0c54580f488bb41575ec013a43801 not found: ID does not exist" containerID="c1ef44dcf181f0deda0c1a5ecce70ff907e0c54580f488bb41575ec013a43801" Jul 19 08:26:52.344799 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:52.344761 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1ef44dcf181f0deda0c1a5ecce70ff907e0c54580f488bb41575ec013a43801"} err="failed to get container status \"c1ef44dcf181f0deda0c1a5ecce70ff907e0c54580f488bb41575ec013a43801\": rpc error: code = NotFound desc = could not find container \"c1ef44dcf181f0deda0c1a5ecce70ff907e0c54580f488bb41575ec013a43801\": container with ID starting with c1ef44dcf181f0deda0c1a5ecce70ff907e0c54580f488bb41575ec013a43801 not found: ID does not exist" Jul 19 08:26:52.348511 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:52.348492 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6eff15b1-openshift-default-54bf58bdf5-lf927"] Jul 19 08:26:52.353356 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:52.353337 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-6eff15b1-openshift-default-54bf58bdf5-lf927"] Jul 19 08:26:52.356691 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:52.356668 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:26:52.424894 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:52.424853 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:26:52.504534 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:52.504483 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:26:52.559688 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:52.559657 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19ae90a7-a58d-43f2-8c19-a93fca3d518f" path="/var/lib/kubelet/pods/19ae90a7-a58d-43f2-8c19-a93fca3d518f/volumes" Jul 19 08:26:52.689491 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:52.689391 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:26:58.236768 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:58.236731 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:26:58.294782 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:26:58.294740 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:27:04.883107 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:04.883065 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:27:05.157422 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:05.157333 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-07a46e45-openshift-default-7c56497d6c-c877v"] Jul 19 08:27:05.157836 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:05.157816 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19ae90a7-a58d-43f2-8c19-a93fca3d518f" containerName="istio-proxy" Jul 19 08:27:05.157939 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:05.157838 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="19ae90a7-a58d-43f2-8c19-a93fca3d518f" containerName="istio-proxy" Jul 19 08:27:05.157939 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:05.157867 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="374d5762-6a34-4049-93f2-924a49b8e40a" containerName="maas-api" Jul 19 08:27:05.157939 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:05.157877 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="374d5762-6a34-4049-93f2-924a49b8e40a" containerName="maas-api" Jul 19 08:27:05.158085 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:05.157985 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="19ae90a7-a58d-43f2-8c19-a93fca3d518f" containerName="istio-proxy" Jul 19 08:27:05.158085 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:05.157999 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="19ae90a7-a58d-43f2-8c19-a93fca3d518f" containerName="istio-proxy" Jul 19 08:27:05.158085 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:05.158007 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="374d5762-6a34-4049-93f2-924a49b8e40a" containerName="maas-api" Jul 19 08:27:05.161239 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:05.161215 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-07a46e45-openshift-default-7c56497d6c-c877v" Jul 19 08:27:05.163781 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:05.163757 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-07a46e45-openshift-default-dockercfg-rqktj\"" Jul 19 08:27:05.163895 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:05.163818 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 19 08:27:05.171540 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:05.171488 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-07a46e45-openshift-default-7c56497d6c-c877v"] Jul 19 08:27:05.182239 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:05.182178 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:27:05.256980 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:05.256940 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c7ba2a4-1abd-49e4-908c-1bae763d00e3-workload-certs\") pod \"e2e-mt-07a46e45-openshift-default-7c56497d6c-c877v\" (UID: \"5c7ba2a4-1abd-49e4-908c-1bae763d00e3\") " pod="openshift-ingress/e2e-mt-07a46e45-openshift-default-7c56497d6c-c877v" Jul 19 08:27:05.257187 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:05.256984 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c7ba2a4-1abd-49e4-908c-1bae763d00e3-istio-envoy\") pod \"e2e-mt-07a46e45-openshift-default-7c56497d6c-c877v\" (UID: \"5c7ba2a4-1abd-49e4-908c-1bae763d00e3\") " pod="openshift-ingress/e2e-mt-07a46e45-openshift-default-7c56497d6c-c877v" Jul 19 08:27:05.257187 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:05.257042 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c7ba2a4-1abd-49e4-908c-1bae763d00e3-istio-token\") pod \"e2e-mt-07a46e45-openshift-default-7c56497d6c-c877v\" (UID: \"5c7ba2a4-1abd-49e4-908c-1bae763d00e3\") " pod="openshift-ingress/e2e-mt-07a46e45-openshift-default-7c56497d6c-c877v" Jul 19 08:27:05.257187 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:05.257126 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c7ba2a4-1abd-49e4-908c-1bae763d00e3-istio-data\") pod \"e2e-mt-07a46e45-openshift-default-7c56497d6c-c877v\" (UID: \"5c7ba2a4-1abd-49e4-908c-1bae763d00e3\") " pod="openshift-ingress/e2e-mt-07a46e45-openshift-default-7c56497d6c-c877v" Jul 19 08:27:05.257187 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:05.257172 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c7ba2a4-1abd-49e4-908c-1bae763d00e3-istiod-ca-cert\") pod \"e2e-mt-07a46e45-openshift-default-7c56497d6c-c877v\" (UID: \"5c7ba2a4-1abd-49e4-908c-1bae763d00e3\") " pod="openshift-ingress/e2e-mt-07a46e45-openshift-default-7c56497d6c-c877v" Jul 19 08:27:05.257419 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:05.257192 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c7ba2a4-1abd-49e4-908c-1bae763d00e3-istio-podinfo\") pod \"e2e-mt-07a46e45-openshift-default-7c56497d6c-c877v\" (UID: \"5c7ba2a4-1abd-49e4-908c-1bae763d00e3\") " pod="openshift-ingress/e2e-mt-07a46e45-openshift-default-7c56497d6c-c877v" Jul 19 08:27:05.257419 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:05.257237 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwsrr\" (UniqueName: \"kubernetes.io/projected/5c7ba2a4-1abd-49e4-908c-1bae763d00e3-kube-api-access-wwsrr\") pod \"e2e-mt-07a46e45-openshift-default-7c56497d6c-c877v\" (UID: \"5c7ba2a4-1abd-49e4-908c-1bae763d00e3\") " pod="openshift-ingress/e2e-mt-07a46e45-openshift-default-7c56497d6c-c877v" Jul 19 08:27:05.257419 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:05.257303 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c7ba2a4-1abd-49e4-908c-1bae763d00e3-workload-socket\") pod \"e2e-mt-07a46e45-openshift-default-7c56497d6c-c877v\" (UID: \"5c7ba2a4-1abd-49e4-908c-1bae763d00e3\") " pod="openshift-ingress/e2e-mt-07a46e45-openshift-default-7c56497d6c-c877v" Jul 19 08:27:05.257419 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:05.257324 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c7ba2a4-1abd-49e4-908c-1bae763d00e3-credential-socket\") pod \"e2e-mt-07a46e45-openshift-default-7c56497d6c-c877v\" (UID: \"5c7ba2a4-1abd-49e4-908c-1bae763d00e3\") " pod="openshift-ingress/e2e-mt-07a46e45-openshift-default-7c56497d6c-c877v" Jul 19 08:27:05.357810 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:05.357768 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c7ba2a4-1abd-49e4-908c-1bae763d00e3-workload-socket\") pod \"e2e-mt-07a46e45-openshift-default-7c56497d6c-c877v\" (UID: \"5c7ba2a4-1abd-49e4-908c-1bae763d00e3\") " pod="openshift-ingress/e2e-mt-07a46e45-openshift-default-7c56497d6c-c877v" Jul 19 08:27:05.357810 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:05.357816 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c7ba2a4-1abd-49e4-908c-1bae763d00e3-credential-socket\") pod \"e2e-mt-07a46e45-openshift-default-7c56497d6c-c877v\" (UID: \"5c7ba2a4-1abd-49e4-908c-1bae763d00e3\") " pod="openshift-ingress/e2e-mt-07a46e45-openshift-default-7c56497d6c-c877v" Jul 19 08:27:05.358085 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:05.357859 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c7ba2a4-1abd-49e4-908c-1bae763d00e3-workload-certs\") pod \"e2e-mt-07a46e45-openshift-default-7c56497d6c-c877v\" (UID: \"5c7ba2a4-1abd-49e4-908c-1bae763d00e3\") " pod="openshift-ingress/e2e-mt-07a46e45-openshift-default-7c56497d6c-c877v" Jul 19 08:27:05.358085 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:05.357883 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c7ba2a4-1abd-49e4-908c-1bae763d00e3-istio-envoy\") pod \"e2e-mt-07a46e45-openshift-default-7c56497d6c-c877v\" (UID: \"5c7ba2a4-1abd-49e4-908c-1bae763d00e3\") " pod="openshift-ingress/e2e-mt-07a46e45-openshift-default-7c56497d6c-c877v" Jul 19 08:27:05.358085 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:05.357942 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c7ba2a4-1abd-49e4-908c-1bae763d00e3-istio-token\") pod \"e2e-mt-07a46e45-openshift-default-7c56497d6c-c877v\" (UID: \"5c7ba2a4-1abd-49e4-908c-1bae763d00e3\") " pod="openshift-ingress/e2e-mt-07a46e45-openshift-default-7c56497d6c-c877v" Jul 19 08:27:05.358085 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:05.357984 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c7ba2a4-1abd-49e4-908c-1bae763d00e3-istio-data\") pod \"e2e-mt-07a46e45-openshift-default-7c56497d6c-c877v\" (UID: \"5c7ba2a4-1abd-49e4-908c-1bae763d00e3\") " pod="openshift-ingress/e2e-mt-07a46e45-openshift-default-7c56497d6c-c877v" Jul 19 08:27:05.358085 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:05.358011 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c7ba2a4-1abd-49e4-908c-1bae763d00e3-istiod-ca-cert\") pod \"e2e-mt-07a46e45-openshift-default-7c56497d6c-c877v\" (UID: \"5c7ba2a4-1abd-49e4-908c-1bae763d00e3\") " pod="openshift-ingress/e2e-mt-07a46e45-openshift-default-7c56497d6c-c877v" Jul 19 08:27:05.358085 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:05.358033 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c7ba2a4-1abd-49e4-908c-1bae763d00e3-istio-podinfo\") pod \"e2e-mt-07a46e45-openshift-default-7c56497d6c-c877v\" (UID: \"5c7ba2a4-1abd-49e4-908c-1bae763d00e3\") " pod="openshift-ingress/e2e-mt-07a46e45-openshift-default-7c56497d6c-c877v" Jul 19 08:27:05.358085 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:05.358056 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwsrr\" (UniqueName: \"kubernetes.io/projected/5c7ba2a4-1abd-49e4-908c-1bae763d00e3-kube-api-access-wwsrr\") pod \"e2e-mt-07a46e45-openshift-default-7c56497d6c-c877v\" (UID: \"5c7ba2a4-1abd-49e4-908c-1bae763d00e3\") " pod="openshift-ingress/e2e-mt-07a46e45-openshift-default-7c56497d6c-c877v" Jul 19 08:27:05.358455 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:05.358291 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c7ba2a4-1abd-49e4-908c-1bae763d00e3-credential-socket\") pod \"e2e-mt-07a46e45-openshift-default-7c56497d6c-c877v\" (UID: \"5c7ba2a4-1abd-49e4-908c-1bae763d00e3\") " pod="openshift-ingress/e2e-mt-07a46e45-openshift-default-7c56497d6c-c877v" Jul 19 08:27:05.358577 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:05.358550 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c7ba2a4-1abd-49e4-908c-1bae763d00e3-workload-socket\") pod \"e2e-mt-07a46e45-openshift-default-7c56497d6c-c877v\" (UID: \"5c7ba2a4-1abd-49e4-908c-1bae763d00e3\") " pod="openshift-ingress/e2e-mt-07a46e45-openshift-default-7c56497d6c-c877v" Jul 19 08:27:05.358772 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:05.358560 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c7ba2a4-1abd-49e4-908c-1bae763d00e3-istio-data\") pod \"e2e-mt-07a46e45-openshift-default-7c56497d6c-c877v\" (UID: \"5c7ba2a4-1abd-49e4-908c-1bae763d00e3\") " pod="openshift-ingress/e2e-mt-07a46e45-openshift-default-7c56497d6c-c877v" Jul 19 08:27:05.358772 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:05.358647 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c7ba2a4-1abd-49e4-908c-1bae763d00e3-workload-certs\") pod \"e2e-mt-07a46e45-openshift-default-7c56497d6c-c877v\" (UID: \"5c7ba2a4-1abd-49e4-908c-1bae763d00e3\") " pod="openshift-ingress/e2e-mt-07a46e45-openshift-default-7c56497d6c-c877v" Jul 19 08:27:05.359003 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:05.358982 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c7ba2a4-1abd-49e4-908c-1bae763d00e3-istiod-ca-cert\") pod \"e2e-mt-07a46e45-openshift-default-7c56497d6c-c877v\" (UID: \"5c7ba2a4-1abd-49e4-908c-1bae763d00e3\") " pod="openshift-ingress/e2e-mt-07a46e45-openshift-default-7c56497d6c-c877v" Jul 19 08:27:05.360977 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:05.360953 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c7ba2a4-1abd-49e4-908c-1bae763d00e3-istio-envoy\") pod \"e2e-mt-07a46e45-openshift-default-7c56497d6c-c877v\" (UID: \"5c7ba2a4-1abd-49e4-908c-1bae763d00e3\") " pod="openshift-ingress/e2e-mt-07a46e45-openshift-default-7c56497d6c-c877v" Jul 19 08:27:05.361068 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:05.360981 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c7ba2a4-1abd-49e4-908c-1bae763d00e3-istio-podinfo\") pod \"e2e-mt-07a46e45-openshift-default-7c56497d6c-c877v\" (UID: \"5c7ba2a4-1abd-49e4-908c-1bae763d00e3\") " pod="openshift-ingress/e2e-mt-07a46e45-openshift-default-7c56497d6c-c877v" Jul 19 08:27:05.365910 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:05.365854 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c7ba2a4-1abd-49e4-908c-1bae763d00e3-istio-token\") pod \"e2e-mt-07a46e45-openshift-default-7c56497d6c-c877v\" (UID: \"5c7ba2a4-1abd-49e4-908c-1bae763d00e3\") " pod="openshift-ingress/e2e-mt-07a46e45-openshift-default-7c56497d6c-c877v" Jul 19 08:27:05.366135 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:05.366110 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwsrr\" (UniqueName: \"kubernetes.io/projected/5c7ba2a4-1abd-49e4-908c-1bae763d00e3-kube-api-access-wwsrr\") pod \"e2e-mt-07a46e45-openshift-default-7c56497d6c-c877v\" (UID: \"5c7ba2a4-1abd-49e4-908c-1bae763d00e3\") " pod="openshift-ingress/e2e-mt-07a46e45-openshift-default-7c56497d6c-c877v" Jul 19 08:27:05.473141 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:05.473101 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-07a46e45-openshift-default-7c56497d6c-c877v" Jul 19 08:27:05.606264 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:05.606211 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:27:05.631549 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:05.631433 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-07a46e45-openshift-default-7c56497d6c-c877v"] Jul 19 08:27:05.634027 ip-10-0-136-70 kubenswrapper[2574]: W0719 08:27:05.633995 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c7ba2a4_1abd_49e4_908c_1bae763d00e3.slice/crio-fc0204ca7d7951dbf800832e8677f2f892605bfbec691a674a74991c594fa9d8 WatchSource:0}: Error finding container fc0204ca7d7951dbf800832e8677f2f892605bfbec691a674a74991c594fa9d8: Status 404 returned error can't find the container with id fc0204ca7d7951dbf800832e8677f2f892605bfbec691a674a74991c594fa9d8 Jul 19 08:27:05.636731 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:05.636673 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 19 08:27:05.636900 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:05.636763 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 19 08:27:05.636900 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:05.636808 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 19 08:27:06.399090 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:06.399048 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-07a46e45-openshift-default-7c56497d6c-c877v" event={"ID":"5c7ba2a4-1abd-49e4-908c-1bae763d00e3","Type":"ContainerStarted","Data":"8ac0072cc65bddbcc1433c27fa76adaead3f79e7cbdfd99b04d32c228d2c5443"} Jul 19 08:27:06.399495 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:06.399108 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-07a46e45-openshift-default-7c56497d6c-c877v" event={"ID":"5c7ba2a4-1abd-49e4-908c-1bae763d00e3","Type":"ContainerStarted","Data":"fc0204ca7d7951dbf800832e8677f2f892605bfbec691a674a74991c594fa9d8"} Jul 19 08:27:06.418257 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:06.418202 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-07a46e45-openshift-default-7c56497d6c-c877v" podStartSLOduration=1.418184387 podStartE2EDuration="1.418184387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 08:27:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:27:06.4175397 +0000 UTC m=+2692.771849304" watchObservedRunningTime="2026-07-19 08:27:06.418184387 +0000 UTC m=+2692.772493988" Jul 19 08:27:06.474140 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:06.474109 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-07a46e45-openshift-default-7c56497d6c-c877v" Jul 19 08:27:06.479590 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:06.479564 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-07a46e45-openshift-default-7c56497d6c-c877v" Jul 19 08:27:07.404666 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:07.404630 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-07a46e45-openshift-default-7c56497d6c-c877v" Jul 19 08:27:07.405857 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:07.405833 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-07a46e45-openshift-default-7c56497d6c-c877v" Jul 19 08:27:08.364530 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:08.364474 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:27:08.440719 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:08.440681 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:27:08.441268 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:08.441031 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-07a46e45-85fb494d8-sqsgd"] Jul 19 08:27:08.441990 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:08.441970 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19ae90a7-a58d-43f2-8c19-a93fca3d518f" containerName="istio-proxy" Jul 19 08:27:08.441990 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:08.441991 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="19ae90a7-a58d-43f2-8c19-a93fca3d518f" containerName="istio-proxy" Jul 19 08:27:08.445348 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:08.445325 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-07a46e45-85fb494d8-sqsgd" Jul 19 08:27:08.447972 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:08.447770 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-07a46e45-serving-cert\"" Jul 19 08:27:08.447972 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:08.447807 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 19 08:27:08.447972 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:08.447871 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-tdbkh\"" Jul 19 08:27:08.448798 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:08.448775 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 19 08:27:08.455773 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:08.455746 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-07a46e45-85fb494d8-sqsgd"] Jul 19 08:27:08.501430 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:08.501394 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:27:08.587925 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:08.587885 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32eee274-b591-439f-826b-d7a276692925-maas-api-tls\") pod \"maas-api-e2e-mt-07a46e45-85fb494d8-sqsgd\" (UID: \"32eee274-b591-439f-826b-d7a276692925\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-07a46e45-85fb494d8-sqsgd" Jul 19 08:27:08.587925 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:08.587926 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32eee274-b591-439f-826b-d7a276692925-openshift-service-ca\") pod \"maas-api-e2e-mt-07a46e45-85fb494d8-sqsgd\" (UID: \"32eee274-b591-439f-826b-d7a276692925\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-07a46e45-85fb494d8-sqsgd" Jul 19 08:27:08.588192 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:08.588075 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gltc\" (UniqueName: \"kubernetes.io/projected/32eee274-b591-439f-826b-d7a276692925-kube-api-access-5gltc\") pod \"maas-api-e2e-mt-07a46e45-85fb494d8-sqsgd\" (UID: \"32eee274-b591-439f-826b-d7a276692925\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-07a46e45-85fb494d8-sqsgd" Jul 19 08:27:08.689188 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:08.689097 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gltc\" (UniqueName: \"kubernetes.io/projected/32eee274-b591-439f-826b-d7a276692925-kube-api-access-5gltc\") pod \"maas-api-e2e-mt-07a46e45-85fb494d8-sqsgd\" (UID: \"32eee274-b591-439f-826b-d7a276692925\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-07a46e45-85fb494d8-sqsgd" Jul 19 08:27:08.689188 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:08.689160 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32eee274-b591-439f-826b-d7a276692925-maas-api-tls\") pod \"maas-api-e2e-mt-07a46e45-85fb494d8-sqsgd\" (UID: \"32eee274-b591-439f-826b-d7a276692925\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-07a46e45-85fb494d8-sqsgd" Jul 19 08:27:08.689188 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:08.689189 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32eee274-b591-439f-826b-d7a276692925-openshift-service-ca\") pod \"maas-api-e2e-mt-07a46e45-85fb494d8-sqsgd\" (UID: \"32eee274-b591-439f-826b-d7a276692925\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-07a46e45-85fb494d8-sqsgd" Jul 19 08:27:08.689423 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:27:08.689329 2574 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-07a46e45-serving-cert: secret "maas-api-e2e-mt-07a46e45-serving-cert" not found Jul 19 08:27:08.689423 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:27:08.689402 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32eee274-b591-439f-826b-d7a276692925-maas-api-tls podName:32eee274-b591-439f-826b-d7a276692925 nodeName:}" failed. No retries permitted until 2026-07-19 08:27:09.189381308 +0000 UTC m=+2695.543690893 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/32eee274-b591-439f-826b-d7a276692925-maas-api-tls") pod "maas-api-e2e-mt-07a46e45-85fb494d8-sqsgd" (UID: "32eee274-b591-439f-826b-d7a276692925") : secret "maas-api-e2e-mt-07a46e45-serving-cert" not found Jul 19 08:27:08.689996 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:08.689977 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32eee274-b591-439f-826b-d7a276692925-openshift-service-ca\") pod \"maas-api-e2e-mt-07a46e45-85fb494d8-sqsgd\" (UID: \"32eee274-b591-439f-826b-d7a276692925\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-07a46e45-85fb494d8-sqsgd" Jul 19 08:27:08.698114 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:08.698090 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gltc\" (UniqueName: \"kubernetes.io/projected/32eee274-b591-439f-826b-d7a276692925-kube-api-access-5gltc\") pod \"maas-api-e2e-mt-07a46e45-85fb494d8-sqsgd\" (UID: \"32eee274-b591-439f-826b-d7a276692925\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-07a46e45-85fb494d8-sqsgd" Jul 19 08:27:08.937374 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:08.937337 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:27:09.033442 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:09.033400 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:27:09.133217 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:09.133177 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:27:09.194435 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:09.194383 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32eee274-b591-439f-826b-d7a276692925-maas-api-tls\") pod \"maas-api-e2e-mt-07a46e45-85fb494d8-sqsgd\" (UID: \"32eee274-b591-439f-826b-d7a276692925\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-07a46e45-85fb494d8-sqsgd" Jul 19 08:27:09.196911 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:09.196892 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32eee274-b591-439f-826b-d7a276692925-maas-api-tls\") pod \"maas-api-e2e-mt-07a46e45-85fb494d8-sqsgd\" (UID: \"32eee274-b591-439f-826b-d7a276692925\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-07a46e45-85fb494d8-sqsgd" Jul 19 08:27:09.359717 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:09.359628 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-07a46e45-85fb494d8-sqsgd" Jul 19 08:27:09.539372 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:09.539346 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-07a46e45-85fb494d8-sqsgd"] Jul 19 08:27:09.541974 ip-10-0-136-70 kubenswrapper[2574]: W0719 08:27:09.541943 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32eee274_b591_439f_826b_d7a276692925.slice/crio-111c2ba7c6fc9f90b65273d7b5d475d7bfb58bd6d7256703f3e5ca3c917f90d8 WatchSource:0}: Error finding container 111c2ba7c6fc9f90b65273d7b5d475d7bfb58bd6d7256703f3e5ca3c917f90d8: Status 404 returned error can't find the container with id 111c2ba7c6fc9f90b65273d7b5d475d7bfb58bd6d7256703f3e5ca3c917f90d8 Jul 19 08:27:10.423101 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:10.423060 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-07a46e45-85fb494d8-sqsgd" event={"ID":"32eee274-b591-439f-826b-d7a276692925","Type":"ContainerStarted","Data":"bf6ab0fb4aad96158712023ae98f1c346d67bd26d7b5040de4ec26e31ee065e2"} Jul 19 08:27:10.423310 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:10.423128 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-07a46e45-85fb494d8-sqsgd" event={"ID":"32eee274-b591-439f-826b-d7a276692925","Type":"ContainerStarted","Data":"111c2ba7c6fc9f90b65273d7b5d475d7bfb58bd6d7256703f3e5ca3c917f90d8"} Jul 19 08:27:10.423310 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:10.423176 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-07a46e45-85fb494d8-sqsgd" Jul 19 08:27:10.440580 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:10.440499 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-07a46e45-85fb494d8-sqsgd" podStartSLOduration=2.138815634 podStartE2EDuration="2.440484437s" totalImagesPullingTime="301.668803ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 08:27:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:27:10.439682887 +0000 UTC m=+2696.793992488" watchObservedRunningTime="2026-07-19 08:27:10.440484437 +0000 UTC m=+2696.794794016" Jul 19 08:27:11.170241 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:11.170203 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 08:27:12.750131 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:12.750087 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:27:14.723866 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:14.723830 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wmhfd_dd68d31d-f763-433a-867b-65b12eabac76/console-operator/1.log" Jul 19 08:27:14.737545 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:14.736987 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wmhfd_dd68d31d-f763-433a-867b-65b12eabac76/console-operator/1.log" Jul 19 08:27:16.434113 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:16.434028 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-07a46e45-85fb494d8-sqsgd" Jul 19 08:27:19.492324 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:19.492288 2574 generic.go:356] "Generic (PLEG): container finished" podID="5c7ba2a4-1abd-49e4-908c-1bae763d00e3" containerID="8ac0072cc65bddbcc1433c27fa76adaead3f79e7cbdfd99b04d32c228d2c5443" exitCode=0 Jul 19 08:27:19.492840 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:19.492379 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-07a46e45-openshift-default-7c56497d6c-c877v" event={"ID":"5c7ba2a4-1abd-49e4-908c-1bae763d00e3","Type":"ContainerDied","Data":"8ac0072cc65bddbcc1433c27fa76adaead3f79e7cbdfd99b04d32c228d2c5443"} Jul 19 08:27:19.492917 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:19.492846 2574 scope.go:117] "RemoveContainer" containerID="8ac0072cc65bddbcc1433c27fa76adaead3f79e7cbdfd99b04d32c228d2c5443" Jul 19 08:27:19.493887 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:19.493858 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 19 08:27:19.493951 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:19.493926 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 19 08:27:19.493989 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:19.493959 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 19 08:27:20.499048 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:20.499008 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-07a46e45-openshift-default-7c56497d6c-c877v" event={"ID":"5c7ba2a4-1abd-49e4-908c-1bae763d00e3","Type":"ContainerStarted","Data":"d5115a9ae97483a144c8e3cad0b4a3b76d1a182a373282cfd476958730e125ad"} Jul 19 08:27:21.473881 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:21.473840 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-07a46e45-openshift-default-7c56497d6c-c877v" Jul 19 08:27:21.475060 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:21.475036 2574 patch_prober.go:28] interesting pod/e2e-mt-07a46e45-openshift-default-7c56497d6c-c877v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Jul 19 08:27:21.475124 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:21.475085 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-07a46e45-openshift-default-7c56497d6c-c877v" podUID="5c7ba2a4-1abd-49e4-908c-1bae763d00e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Jul 19 08:27:22.405157 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:22.405124 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-07a46e45-openshift-default-7c56497d6c-c877v" Jul 19 08:27:22.474112 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:22.474068 2574 patch_prober.go:28] interesting pod/e2e-mt-07a46e45-openshift-default-7c56497d6c-c877v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Jul 19 08:27:22.474301 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:22.474130 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-07a46e45-openshift-default-7c56497d6c-c877v" podUID="5c7ba2a4-1abd-49e4-908c-1bae763d00e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Jul 19 08:27:23.473790 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:23.473743 2574 patch_prober.go:28] interesting pod/e2e-mt-07a46e45-openshift-default-7c56497d6c-c877v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Jul 19 08:27:23.474199 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:23.473817 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-07a46e45-openshift-default-7c56497d6c-c877v" podUID="5c7ba2a4-1abd-49e4-908c-1bae763d00e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Jul 19 08:27:24.474064 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:24.474022 2574 patch_prober.go:28] interesting pod/e2e-mt-07a46e45-openshift-default-7c56497d6c-c877v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Jul 19 08:27:24.474573 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:24.474101 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-07a46e45-openshift-default-7c56497d6c-c877v" podUID="5c7ba2a4-1abd-49e4-908c-1bae763d00e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Jul 19 08:27:25.477206 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:25.477173 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-07a46e45-openshift-default-7c56497d6c-c877v" Jul 19 08:27:25.478205 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:25.478187 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-07a46e45-openshift-default-7c56497d6c-c877v" Jul 19 08:27:35.410156 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:35.410118 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:27:38.566406 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:38.566367 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 08:27:44.072773 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:27:44.072725 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:28:00.866419 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:28:00.866374 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 08:28:16.703172 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:28:16.703134 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-07a46e45-85fb494d8-sqsgd"] Jul 19 08:28:16.703708 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:28:16.703423 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-07a46e45-85fb494d8-sqsgd" podUID="32eee274-b591-439f-826b-d7a276692925" containerName="maas-api" containerID="cri-o://bf6ab0fb4aad96158712023ae98f1c346d67bd26d7b5040de4ec26e31ee065e2" gracePeriod=30 Jul 19 08:28:16.730715 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:28:16.730684 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:28:16.956455 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:28:16.956376 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-07a46e45-85fb494d8-sqsgd" Jul 19 08:28:17.024935 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:28:17.024897 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32eee274-b591-439f-826b-d7a276692925-openshift-service-ca\") pod \"32eee274-b591-439f-826b-d7a276692925\" (UID: \"32eee274-b591-439f-826b-d7a276692925\") " Jul 19 08:28:17.025129 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:28:17.024954 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5gltc\" (UniqueName: \"kubernetes.io/projected/32eee274-b591-439f-826b-d7a276692925-kube-api-access-5gltc\") pod \"32eee274-b591-439f-826b-d7a276692925\" (UID: \"32eee274-b591-439f-826b-d7a276692925\") " Jul 19 08:28:17.025129 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:28:17.024987 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32eee274-b591-439f-826b-d7a276692925-maas-api-tls\") pod \"32eee274-b591-439f-826b-d7a276692925\" (UID: \"32eee274-b591-439f-826b-d7a276692925\") " Jul 19 08:28:17.025361 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:28:17.025336 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32eee274-b591-439f-826b-d7a276692925-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "32eee274-b591-439f-826b-d7a276692925" (UID: "32eee274-b591-439f-826b-d7a276692925"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 08:28:17.027129 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:28:17.027095 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32eee274-b591-439f-826b-d7a276692925-kube-api-access-5gltc" (OuterVolumeSpecName: "kube-api-access-5gltc") pod "32eee274-b591-439f-826b-d7a276692925" (UID: "32eee274-b591-439f-826b-d7a276692925"). InnerVolumeSpecName "kube-api-access-5gltc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 08:28:17.027240 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:28:17.027178 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32eee274-b591-439f-826b-d7a276692925-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "32eee274-b591-439f-826b-d7a276692925" (UID: "32eee274-b591-439f-826b-d7a276692925"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 08:28:17.126176 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:28:17.126127 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32eee274-b591-439f-826b-d7a276692925-openshift-service-ca\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:28:17.126176 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:28:17.126171 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gltc\" (UniqueName: \"kubernetes.io/projected/32eee274-b591-439f-826b-d7a276692925-kube-api-access-5gltc\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:28:17.126176 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:28:17.126186 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32eee274-b591-439f-826b-d7a276692925-maas-api-tls\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:28:17.794159 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:28:17.794122 2574 generic.go:356] "Generic (PLEG): container finished" podID="32eee274-b591-439f-826b-d7a276692925" containerID="bf6ab0fb4aad96158712023ae98f1c346d67bd26d7b5040de4ec26e31ee065e2" exitCode=0 Jul 19 08:28:17.794648 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:28:17.794178 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-07a46e45-85fb494d8-sqsgd" Jul 19 08:28:17.794648 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:28:17.794212 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-07a46e45-85fb494d8-sqsgd" event={"ID":"32eee274-b591-439f-826b-d7a276692925","Type":"ContainerDied","Data":"bf6ab0fb4aad96158712023ae98f1c346d67bd26d7b5040de4ec26e31ee065e2"} Jul 19 08:28:17.794648 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:28:17.794268 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-07a46e45-85fb494d8-sqsgd" event={"ID":"32eee274-b591-439f-826b-d7a276692925","Type":"ContainerDied","Data":"111c2ba7c6fc9f90b65273d7b5d475d7bfb58bd6d7256703f3e5ca3c917f90d8"} Jul 19 08:28:17.794648 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:28:17.794291 2574 scope.go:117] "RemoveContainer" containerID="bf6ab0fb4aad96158712023ae98f1c346d67bd26d7b5040de4ec26e31ee065e2" Jul 19 08:28:17.803918 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:28:17.803903 2574 scope.go:117] "RemoveContainer" containerID="bf6ab0fb4aad96158712023ae98f1c346d67bd26d7b5040de4ec26e31ee065e2" Jul 19 08:28:17.804193 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:28:17.804174 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf6ab0fb4aad96158712023ae98f1c346d67bd26d7b5040de4ec26e31ee065e2\": container with ID starting with bf6ab0fb4aad96158712023ae98f1c346d67bd26d7b5040de4ec26e31ee065e2 not found: ID does not exist" containerID="bf6ab0fb4aad96158712023ae98f1c346d67bd26d7b5040de4ec26e31ee065e2" Jul 19 08:28:17.804245 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:28:17.804202 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf6ab0fb4aad96158712023ae98f1c346d67bd26d7b5040de4ec26e31ee065e2"} err="failed to get container status \"bf6ab0fb4aad96158712023ae98f1c346d67bd26d7b5040de4ec26e31ee065e2\": rpc error: code = NotFound desc = could not find container \"bf6ab0fb4aad96158712023ae98f1c346d67bd26d7b5040de4ec26e31ee065e2\": container with ID starting with bf6ab0fb4aad96158712023ae98f1c346d67bd26d7b5040de4ec26e31ee065e2 not found: ID does not exist" Jul 19 08:28:17.815164 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:28:17.815139 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-07a46e45-85fb494d8-sqsgd"] Jul 19 08:28:17.819066 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:28:17.819043 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-07a46e45-85fb494d8-sqsgd"] Jul 19 08:28:18.560132 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:28:18.560102 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32eee274-b591-439f-826b-d7a276692925" path="/var/lib/kubelet/pods/32eee274-b591-439f-826b-d7a276692925/volumes" Jul 19 08:28:20.968292 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:28:20.968253 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 08:28:21.358241 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:28:21.358150 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:28:26.378104 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:28:26.378066 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:28:31.553687 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:28:31.553651 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:28:31.600066 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:28:31.600030 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:28:51.861311 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:28:51.861219 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:28:51.959867 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:28:51.959821 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:29:02.564605 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:29:02.564567 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:29:03.677688 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:29:03.677648 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:29:09.059787 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:29:09.059742 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:29:09.151837 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:29:09.151798 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:29:13.470506 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:29:13.470465 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-07a46e45-openshift-default-7c56497d6c-c877v"] Jul 19 08:29:13.470964 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:29:13.470794 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-07a46e45-openshift-default-7c56497d6c-c877v" podUID="5c7ba2a4-1abd-49e4-908c-1bae763d00e3" containerName="istio-proxy" containerID="cri-o://d5115a9ae97483a144c8e3cad0b4a3b76d1a182a373282cfd476958730e125ad" gracePeriod=30 Jul 19 08:29:13.484963 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:29:13.484934 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:29:13.716607 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:29:13.716566 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:29:14.723865 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:29:14.723824 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:29:14.823790 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:29:14.823749 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:29:14.923274 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:29:14.923240 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:29:15.619945 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:29:15.619904 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:29:15.718813 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:29:15.718777 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:29:15.818323 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:29:15.818281 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:29:17.125093 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:29:17.125050 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:29:18.082548 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:29:18.082480 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 08:29:18.736242 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:29:18.736166 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-07a46e45-openshift-default-7c56497d6c-c877v" Jul 19 08:29:18.769682 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:29:18.769644 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c7ba2a4-1abd-49e4-908c-1bae763d00e3-istio-podinfo\") pod \"5c7ba2a4-1abd-49e4-908c-1bae763d00e3\" (UID: \"5c7ba2a4-1abd-49e4-908c-1bae763d00e3\") " Jul 19 08:29:18.769682 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:29:18.769696 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c7ba2a4-1abd-49e4-908c-1bae763d00e3-istiod-ca-cert\") pod \"5c7ba2a4-1abd-49e4-908c-1bae763d00e3\" (UID: \"5c7ba2a4-1abd-49e4-908c-1bae763d00e3\") " Jul 19 08:29:18.769967 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:29:18.769723 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c7ba2a4-1abd-49e4-908c-1bae763d00e3-istio-envoy\") pod \"5c7ba2a4-1abd-49e4-908c-1bae763d00e3\" (UID: \"5c7ba2a4-1abd-49e4-908c-1bae763d00e3\") " Jul 19 08:29:18.769967 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:29:18.769767 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c7ba2a4-1abd-49e4-908c-1bae763d00e3-workload-socket\") pod \"5c7ba2a4-1abd-49e4-908c-1bae763d00e3\" (UID: \"5c7ba2a4-1abd-49e4-908c-1bae763d00e3\") " Jul 19 08:29:18.769967 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:29:18.769788 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wwsrr\" (UniqueName: \"kubernetes.io/projected/5c7ba2a4-1abd-49e4-908c-1bae763d00e3-kube-api-access-wwsrr\") pod \"5c7ba2a4-1abd-49e4-908c-1bae763d00e3\" (UID: \"5c7ba2a4-1abd-49e4-908c-1bae763d00e3\") " Jul 19 08:29:18.769967 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:29:18.769866 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c7ba2a4-1abd-49e4-908c-1bae763d00e3-workload-certs\") pod \"5c7ba2a4-1abd-49e4-908c-1bae763d00e3\" (UID: \"5c7ba2a4-1abd-49e4-908c-1bae763d00e3\") " Jul 19 08:29:18.769967 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:29:18.769892 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c7ba2a4-1abd-49e4-908c-1bae763d00e3-istio-token\") pod \"5c7ba2a4-1abd-49e4-908c-1bae763d00e3\" (UID: \"5c7ba2a4-1abd-49e4-908c-1bae763d00e3\") " Jul 19 08:29:18.769967 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:29:18.769915 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c7ba2a4-1abd-49e4-908c-1bae763d00e3-istio-data\") pod \"5c7ba2a4-1abd-49e4-908c-1bae763d00e3\" (UID: \"5c7ba2a4-1abd-49e4-908c-1bae763d00e3\") " Jul 19 08:29:18.769967 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:29:18.769939 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c7ba2a4-1abd-49e4-908c-1bae763d00e3-credential-socket\") pod \"5c7ba2a4-1abd-49e4-908c-1bae763d00e3\" (UID: \"5c7ba2a4-1abd-49e4-908c-1bae763d00e3\") " Jul 19 08:29:18.770342 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:29:18.770098 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c7ba2a4-1abd-49e4-908c-1bae763d00e3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5c7ba2a4-1abd-49e4-908c-1bae763d00e3" (UID: "5c7ba2a4-1abd-49e4-908c-1bae763d00e3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 08:29:18.770342 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:29:18.770161 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c7ba2a4-1abd-49e4-908c-1bae763d00e3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5c7ba2a4-1abd-49e4-908c-1bae763d00e3" (UID: "5c7ba2a4-1abd-49e4-908c-1bae763d00e3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 08:29:18.770342 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:29:18.770289 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c7ba2a4-1abd-49e4-908c-1bae763d00e3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5c7ba2a4-1abd-49e4-908c-1bae763d00e3" (UID: "5c7ba2a4-1abd-49e4-908c-1bae763d00e3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 08:29:18.770342 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:29:18.770315 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c7ba2a4-1abd-49e4-908c-1bae763d00e3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5c7ba2a4-1abd-49e4-908c-1bae763d00e3" (UID: "5c7ba2a4-1abd-49e4-908c-1bae763d00e3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 08:29:18.770598 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:29:18.770340 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c7ba2a4-1abd-49e4-908c-1bae763d00e3-istiod-ca-cert\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:29:18.770598 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:29:18.770361 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c7ba2a4-1abd-49e4-908c-1bae763d00e3-workload-socket\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:29:18.771283 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:29:18.771250 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c7ba2a4-1abd-49e4-908c-1bae763d00e3-istio-data" (OuterVolumeSpecName: "istio-data") pod "5c7ba2a4-1abd-49e4-908c-1bae763d00e3" (UID: "5c7ba2a4-1abd-49e4-908c-1bae763d00e3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 08:29:18.772403 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:29:18.772381 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5c7ba2a4-1abd-49e4-908c-1bae763d00e3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5c7ba2a4-1abd-49e4-908c-1bae763d00e3" (UID: "5c7ba2a4-1abd-49e4-908c-1bae763d00e3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 19 08:29:18.772547 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:29:18.772465 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c7ba2a4-1abd-49e4-908c-1bae763d00e3-istio-token" (OuterVolumeSpecName: "istio-token") pod "5c7ba2a4-1abd-49e4-908c-1bae763d00e3" (UID: "5c7ba2a4-1abd-49e4-908c-1bae763d00e3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 08:29:18.772819 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:29:18.772798 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c7ba2a4-1abd-49e4-908c-1bae763d00e3-kube-api-access-wwsrr" (OuterVolumeSpecName: "kube-api-access-wwsrr") pod "5c7ba2a4-1abd-49e4-908c-1bae763d00e3" (UID: "5c7ba2a4-1abd-49e4-908c-1bae763d00e3"). InnerVolumeSpecName "kube-api-access-wwsrr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 08:29:18.773192 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:29:18.773172 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c7ba2a4-1abd-49e4-908c-1bae763d00e3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5c7ba2a4-1abd-49e4-908c-1bae763d00e3" (UID: "5c7ba2a4-1abd-49e4-908c-1bae763d00e3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 08:29:18.871228 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:29:18.871187 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwsrr\" (UniqueName: \"kubernetes.io/projected/5c7ba2a4-1abd-49e4-908c-1bae763d00e3-kube-api-access-wwsrr\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:29:18.871228 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:29:18.871225 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c7ba2a4-1abd-49e4-908c-1bae763d00e3-workload-certs\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:29:18.871228 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:29:18.871237 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c7ba2a4-1abd-49e4-908c-1bae763d00e3-istio-token\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:29:18.871482 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:29:18.871250 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c7ba2a4-1abd-49e4-908c-1bae763d00e3-istio-data\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:29:18.871482 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:29:18.871261 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c7ba2a4-1abd-49e4-908c-1bae763d00e3-credential-socket\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:29:18.871482 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:29:18.871272 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c7ba2a4-1abd-49e4-908c-1bae763d00e3-istio-podinfo\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:29:18.871482 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:29:18.871282 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c7ba2a4-1abd-49e4-908c-1bae763d00e3-istio-envoy\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:29:19.095784 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:29:19.095695 2574 generic.go:356] "Generic (PLEG): container finished" podID="5c7ba2a4-1abd-49e4-908c-1bae763d00e3" containerID="d5115a9ae97483a144c8e3cad0b4a3b76d1a182a373282cfd476958730e125ad" exitCode=0 Jul 19 08:29:19.095784 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:29:19.095745 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-07a46e45-openshift-default-7c56497d6c-c877v" Jul 19 08:29:19.095784 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:29:19.095759 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-07a46e45-openshift-default-7c56497d6c-c877v" event={"ID":"5c7ba2a4-1abd-49e4-908c-1bae763d00e3","Type":"ContainerDied","Data":"d5115a9ae97483a144c8e3cad0b4a3b76d1a182a373282cfd476958730e125ad"} Jul 19 08:29:19.096035 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:29:19.095830 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-07a46e45-openshift-default-7c56497d6c-c877v" event={"ID":"5c7ba2a4-1abd-49e4-908c-1bae763d00e3","Type":"ContainerDied","Data":"fc0204ca7d7951dbf800832e8677f2f892605bfbec691a674a74991c594fa9d8"} Jul 19 08:29:19.096035 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:29:19.095853 2574 scope.go:117] "RemoveContainer" containerID="d5115a9ae97483a144c8e3cad0b4a3b76d1a182a373282cfd476958730e125ad" Jul 19 08:29:19.105996 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:29:19.105974 2574 scope.go:117] "RemoveContainer" containerID="8ac0072cc65bddbcc1433c27fa76adaead3f79e7cbdfd99b04d32c228d2c5443" Jul 19 08:29:19.114129 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:29:19.114107 2574 scope.go:117] "RemoveContainer" containerID="d5115a9ae97483a144c8e3cad0b4a3b76d1a182a373282cfd476958730e125ad" Jul 19 08:29:19.114411 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:29:19.114388 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5115a9ae97483a144c8e3cad0b4a3b76d1a182a373282cfd476958730e125ad\": container with ID starting with d5115a9ae97483a144c8e3cad0b4a3b76d1a182a373282cfd476958730e125ad not found: ID does not exist" containerID="d5115a9ae97483a144c8e3cad0b4a3b76d1a182a373282cfd476958730e125ad" Jul 19 08:29:19.114456 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:29:19.114420 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5115a9ae97483a144c8e3cad0b4a3b76d1a182a373282cfd476958730e125ad"} err="failed to get container status \"d5115a9ae97483a144c8e3cad0b4a3b76d1a182a373282cfd476958730e125ad\": rpc error: code = NotFound desc = could not find container \"d5115a9ae97483a144c8e3cad0b4a3b76d1a182a373282cfd476958730e125ad\": container with ID starting with d5115a9ae97483a144c8e3cad0b4a3b76d1a182a373282cfd476958730e125ad not found: ID does not exist" Jul 19 08:29:19.114456 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:29:19.114439 2574 scope.go:117] "RemoveContainer" containerID="8ac0072cc65bddbcc1433c27fa76adaead3f79e7cbdfd99b04d32c228d2c5443" Jul 19 08:29:19.114718 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:29:19.114699 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ac0072cc65bddbcc1433c27fa76adaead3f79e7cbdfd99b04d32c228d2c5443\": container with ID starting with 8ac0072cc65bddbcc1433c27fa76adaead3f79e7cbdfd99b04d32c228d2c5443 not found: ID does not exist" containerID="8ac0072cc65bddbcc1433c27fa76adaead3f79e7cbdfd99b04d32c228d2c5443" Jul 19 08:29:19.114780 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:29:19.114722 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ac0072cc65bddbcc1433c27fa76adaead3f79e7cbdfd99b04d32c228d2c5443"} err="failed to get container status \"8ac0072cc65bddbcc1433c27fa76adaead3f79e7cbdfd99b04d32c228d2c5443\": rpc error: code = NotFound desc = could not find container \"8ac0072cc65bddbcc1433c27fa76adaead3f79e7cbdfd99b04d32c228d2c5443\": container with ID starting with 8ac0072cc65bddbcc1433c27fa76adaead3f79e7cbdfd99b04d32c228d2c5443 not found: ID does not exist" Jul 19 08:29:19.118214 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:29:19.118191 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-07a46e45-openshift-default-7c56497d6c-c877v"] Jul 19 08:29:19.121810 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:29:19.121786 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-07a46e45-openshift-default-7c56497d6c-c877v"] Jul 19 08:29:20.558740 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:29:20.558698 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c7ba2a4-1abd-49e4-908c-1bae763d00e3" path="/var/lib/kubelet/pods/5c7ba2a4-1abd-49e4-908c-1bae763d00e3/volumes" Jul 19 08:29:22.677321 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:29:22.677282 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:29:22.737544 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:29:22.737503 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:29:23.350077 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:29:23.350040 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:29:28.901992 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:29:28.901943 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:29:29.011790 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:29:29.011747 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:29:29.107153 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:29:29.107114 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:29:30.871970 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:29:30.871918 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 08:29:39.669432 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:29:39.669392 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:29:40.021433 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:29:40.021395 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:30:00.973506 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:00.973463 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:30:06.809181 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:06.809062 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:30:06.888412 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:06.888372 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:30:07.177314 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:07.177191 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:30:07.432069 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:07.431976 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-49f9c5-openshift-default-777b858b69-gzbsz"] Jul 19 08:30:07.432601 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:07.432577 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32eee274-b591-439f-826b-d7a276692925" containerName="maas-api" Jul 19 08:30:07.432601 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:07.432600 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="32eee274-b591-439f-826b-d7a276692925" containerName="maas-api" Jul 19 08:30:07.432825 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:07.432624 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c7ba2a4-1abd-49e4-908c-1bae763d00e3" containerName="istio-proxy" Jul 19 08:30:07.432825 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:07.432667 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c7ba2a4-1abd-49e4-908c-1bae763d00e3" containerName="istio-proxy" Jul 19 08:30:07.432825 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:07.432682 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c7ba2a4-1abd-49e4-908c-1bae763d00e3" containerName="istio-proxy" Jul 19 08:30:07.432825 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:07.432691 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c7ba2a4-1abd-49e4-908c-1bae763d00e3" containerName="istio-proxy" Jul 19 08:30:07.433032 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:07.432838 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c7ba2a4-1abd-49e4-908c-1bae763d00e3" containerName="istio-proxy" Jul 19 08:30:07.433032 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:07.432853 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c7ba2a4-1abd-49e4-908c-1bae763d00e3" containerName="istio-proxy" Jul 19 08:30:07.433032 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:07.432867 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="32eee274-b591-439f-826b-d7a276692925" containerName="maas-api" Jul 19 08:30:07.443901 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:07.443871 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-49f9c5-openshift-default-777b858b69-gzbsz" Jul 19 08:30:07.446676 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:07.446649 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-49f9c5-openshift-default-dockercfg-hsqp5\"" Jul 19 08:30:07.447126 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:07.447101 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 19 08:30:07.447625 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:07.447597 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-49f9c5-openshift-default-777b858b69-gzbsz"] Jul 19 08:30:07.455100 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:07.455068 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:30:07.482606 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:07.482571 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cf877b7-40dd-4010-9d0e-6c902a6c6ca9-credential-socket\") pod \"e2e-inf-a-49f9c5-openshift-default-777b858b69-gzbsz\" (UID: \"1cf877b7-40dd-4010-9d0e-6c902a6c6ca9\") " pod="openshift-ingress/e2e-inf-a-49f9c5-openshift-default-777b858b69-gzbsz" Jul 19 08:30:07.482606 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:07.482612 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cf877b7-40dd-4010-9d0e-6c902a6c6ca9-istio-token\") pod \"e2e-inf-a-49f9c5-openshift-default-777b858b69-gzbsz\" (UID: \"1cf877b7-40dd-4010-9d0e-6c902a6c6ca9\") " pod="openshift-ingress/e2e-inf-a-49f9c5-openshift-default-777b858b69-gzbsz" Jul 19 08:30:07.482882 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:07.482640 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cf877b7-40dd-4010-9d0e-6c902a6c6ca9-istiod-ca-cert\") pod \"e2e-inf-a-49f9c5-openshift-default-777b858b69-gzbsz\" (UID: \"1cf877b7-40dd-4010-9d0e-6c902a6c6ca9\") " pod="openshift-ingress/e2e-inf-a-49f9c5-openshift-default-777b858b69-gzbsz" Jul 19 08:30:07.482882 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:07.482687 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cf877b7-40dd-4010-9d0e-6c902a6c6ca9-istio-envoy\") pod \"e2e-inf-a-49f9c5-openshift-default-777b858b69-gzbsz\" (UID: \"1cf877b7-40dd-4010-9d0e-6c902a6c6ca9\") " pod="openshift-ingress/e2e-inf-a-49f9c5-openshift-default-777b858b69-gzbsz" Jul 19 08:30:07.482882 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:07.482725 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cf877b7-40dd-4010-9d0e-6c902a6c6ca9-workload-certs\") pod \"e2e-inf-a-49f9c5-openshift-default-777b858b69-gzbsz\" (UID: \"1cf877b7-40dd-4010-9d0e-6c902a6c6ca9\") " pod="openshift-ingress/e2e-inf-a-49f9c5-openshift-default-777b858b69-gzbsz" Jul 19 08:30:07.482882 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:07.482748 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cf877b7-40dd-4010-9d0e-6c902a6c6ca9-istio-data\") pod \"e2e-inf-a-49f9c5-openshift-default-777b858b69-gzbsz\" (UID: \"1cf877b7-40dd-4010-9d0e-6c902a6c6ca9\") " pod="openshift-ingress/e2e-inf-a-49f9c5-openshift-default-777b858b69-gzbsz" Jul 19 08:30:07.482882 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:07.482766 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cf877b7-40dd-4010-9d0e-6c902a6c6ca9-istio-podinfo\") pod \"e2e-inf-a-49f9c5-openshift-default-777b858b69-gzbsz\" (UID: \"1cf877b7-40dd-4010-9d0e-6c902a6c6ca9\") " pod="openshift-ingress/e2e-inf-a-49f9c5-openshift-default-777b858b69-gzbsz" Jul 19 08:30:07.482882 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:07.482824 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cf877b7-40dd-4010-9d0e-6c902a6c6ca9-workload-socket\") pod \"e2e-inf-a-49f9c5-openshift-default-777b858b69-gzbsz\" (UID: \"1cf877b7-40dd-4010-9d0e-6c902a6c6ca9\") " pod="openshift-ingress/e2e-inf-a-49f9c5-openshift-default-777b858b69-gzbsz" Jul 19 08:30:07.483203 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:07.482903 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wf92\" (UniqueName: \"kubernetes.io/projected/1cf877b7-40dd-4010-9d0e-6c902a6c6ca9-kube-api-access-5wf92\") pod \"e2e-inf-a-49f9c5-openshift-default-777b858b69-gzbsz\" (UID: \"1cf877b7-40dd-4010-9d0e-6c902a6c6ca9\") " pod="openshift-ingress/e2e-inf-a-49f9c5-openshift-default-777b858b69-gzbsz" Jul 19 08:30:07.583430 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:07.583397 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wf92\" (UniqueName: \"kubernetes.io/projected/1cf877b7-40dd-4010-9d0e-6c902a6c6ca9-kube-api-access-5wf92\") pod \"e2e-inf-a-49f9c5-openshift-default-777b858b69-gzbsz\" (UID: \"1cf877b7-40dd-4010-9d0e-6c902a6c6ca9\") " pod="openshift-ingress/e2e-inf-a-49f9c5-openshift-default-777b858b69-gzbsz" Jul 19 08:30:07.583636 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:07.583479 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cf877b7-40dd-4010-9d0e-6c902a6c6ca9-credential-socket\") pod \"e2e-inf-a-49f9c5-openshift-default-777b858b69-gzbsz\" (UID: \"1cf877b7-40dd-4010-9d0e-6c902a6c6ca9\") " pod="openshift-ingress/e2e-inf-a-49f9c5-openshift-default-777b858b69-gzbsz" Jul 19 08:30:07.583636 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:07.583500 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cf877b7-40dd-4010-9d0e-6c902a6c6ca9-istio-token\") pod \"e2e-inf-a-49f9c5-openshift-default-777b858b69-gzbsz\" (UID: \"1cf877b7-40dd-4010-9d0e-6c902a6c6ca9\") " pod="openshift-ingress/e2e-inf-a-49f9c5-openshift-default-777b858b69-gzbsz" Jul 19 08:30:07.583636 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:07.583530 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cf877b7-40dd-4010-9d0e-6c902a6c6ca9-istiod-ca-cert\") pod \"e2e-inf-a-49f9c5-openshift-default-777b858b69-gzbsz\" (UID: \"1cf877b7-40dd-4010-9d0e-6c902a6c6ca9\") " pod="openshift-ingress/e2e-inf-a-49f9c5-openshift-default-777b858b69-gzbsz" Jul 19 08:30:07.583636 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:07.583609 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cf877b7-40dd-4010-9d0e-6c902a6c6ca9-istio-envoy\") pod \"e2e-inf-a-49f9c5-openshift-default-777b858b69-gzbsz\" (UID: \"1cf877b7-40dd-4010-9d0e-6c902a6c6ca9\") " pod="openshift-ingress/e2e-inf-a-49f9c5-openshift-default-777b858b69-gzbsz" Jul 19 08:30:07.583932 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:07.583660 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cf877b7-40dd-4010-9d0e-6c902a6c6ca9-workload-certs\") pod \"e2e-inf-a-49f9c5-openshift-default-777b858b69-gzbsz\" (UID: \"1cf877b7-40dd-4010-9d0e-6c902a6c6ca9\") " pod="openshift-ingress/e2e-inf-a-49f9c5-openshift-default-777b858b69-gzbsz" Jul 19 08:30:07.583932 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:07.583694 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cf877b7-40dd-4010-9d0e-6c902a6c6ca9-istio-data\") pod \"e2e-inf-a-49f9c5-openshift-default-777b858b69-gzbsz\" (UID: \"1cf877b7-40dd-4010-9d0e-6c902a6c6ca9\") " pod="openshift-ingress/e2e-inf-a-49f9c5-openshift-default-777b858b69-gzbsz" Jul 19 08:30:07.583932 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:07.583718 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cf877b7-40dd-4010-9d0e-6c902a6c6ca9-istio-podinfo\") pod \"e2e-inf-a-49f9c5-openshift-default-777b858b69-gzbsz\" (UID: \"1cf877b7-40dd-4010-9d0e-6c902a6c6ca9\") " pod="openshift-ingress/e2e-inf-a-49f9c5-openshift-default-777b858b69-gzbsz" Jul 19 08:30:07.583932 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:07.583765 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cf877b7-40dd-4010-9d0e-6c902a6c6ca9-workload-socket\") pod \"e2e-inf-a-49f9c5-openshift-default-777b858b69-gzbsz\" (UID: \"1cf877b7-40dd-4010-9d0e-6c902a6c6ca9\") " pod="openshift-ingress/e2e-inf-a-49f9c5-openshift-default-777b858b69-gzbsz" Jul 19 08:30:07.583932 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:07.583921 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cf877b7-40dd-4010-9d0e-6c902a6c6ca9-credential-socket\") pod \"e2e-inf-a-49f9c5-openshift-default-777b858b69-gzbsz\" (UID: \"1cf877b7-40dd-4010-9d0e-6c902a6c6ca9\") " pod="openshift-ingress/e2e-inf-a-49f9c5-openshift-default-777b858b69-gzbsz" Jul 19 08:30:07.584206 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:07.584152 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cf877b7-40dd-4010-9d0e-6c902a6c6ca9-workload-certs\") pod \"e2e-inf-a-49f9c5-openshift-default-777b858b69-gzbsz\" (UID: \"1cf877b7-40dd-4010-9d0e-6c902a6c6ca9\") " pod="openshift-ingress/e2e-inf-a-49f9c5-openshift-default-777b858b69-gzbsz" Jul 19 08:30:07.584296 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:07.584277 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cf877b7-40dd-4010-9d0e-6c902a6c6ca9-workload-socket\") pod \"e2e-inf-a-49f9c5-openshift-default-777b858b69-gzbsz\" (UID: \"1cf877b7-40dd-4010-9d0e-6c902a6c6ca9\") " pod="openshift-ingress/e2e-inf-a-49f9c5-openshift-default-777b858b69-gzbsz" Jul 19 08:30:07.584379 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:07.584346 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cf877b7-40dd-4010-9d0e-6c902a6c6ca9-istio-data\") pod \"e2e-inf-a-49f9c5-openshift-default-777b858b69-gzbsz\" (UID: \"1cf877b7-40dd-4010-9d0e-6c902a6c6ca9\") " pod="openshift-ingress/e2e-inf-a-49f9c5-openshift-default-777b858b69-gzbsz" Jul 19 08:30:07.584379 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:07.584373 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cf877b7-40dd-4010-9d0e-6c902a6c6ca9-istiod-ca-cert\") pod \"e2e-inf-a-49f9c5-openshift-default-777b858b69-gzbsz\" (UID: \"1cf877b7-40dd-4010-9d0e-6c902a6c6ca9\") " pod="openshift-ingress/e2e-inf-a-49f9c5-openshift-default-777b858b69-gzbsz" Jul 19 08:30:07.585895 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:07.585863 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cf877b7-40dd-4010-9d0e-6c902a6c6ca9-istio-envoy\") pod \"e2e-inf-a-49f9c5-openshift-default-777b858b69-gzbsz\" (UID: \"1cf877b7-40dd-4010-9d0e-6c902a6c6ca9\") " pod="openshift-ingress/e2e-inf-a-49f9c5-openshift-default-777b858b69-gzbsz" Jul 19 08:30:07.586343 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:07.586319 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cf877b7-40dd-4010-9d0e-6c902a6c6ca9-istio-podinfo\") pod \"e2e-inf-a-49f9c5-openshift-default-777b858b69-gzbsz\" (UID: \"1cf877b7-40dd-4010-9d0e-6c902a6c6ca9\") " pod="openshift-ingress/e2e-inf-a-49f9c5-openshift-default-777b858b69-gzbsz" Jul 19 08:30:07.591193 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:07.591169 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cf877b7-40dd-4010-9d0e-6c902a6c6ca9-istio-token\") pod \"e2e-inf-a-49f9c5-openshift-default-777b858b69-gzbsz\" (UID: \"1cf877b7-40dd-4010-9d0e-6c902a6c6ca9\") " pod="openshift-ingress/e2e-inf-a-49f9c5-openshift-default-777b858b69-gzbsz" Jul 19 08:30:07.591289 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:07.591232 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wf92\" (UniqueName: \"kubernetes.io/projected/1cf877b7-40dd-4010-9d0e-6c902a6c6ca9-kube-api-access-5wf92\") pod \"e2e-inf-a-49f9c5-openshift-default-777b858b69-gzbsz\" (UID: \"1cf877b7-40dd-4010-9d0e-6c902a6c6ca9\") " pod="openshift-ingress/e2e-inf-a-49f9c5-openshift-default-777b858b69-gzbsz" Jul 19 08:30:07.757196 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:07.757154 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-49f9c5-openshift-default-777b858b69-gzbsz" Jul 19 08:30:07.901385 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:07.901357 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-49f9c5-openshift-default-777b858b69-gzbsz"] Jul 19 08:30:07.905002 ip-10-0-136-70 kubenswrapper[2574]: W0719 08:30:07.904972 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cf877b7_40dd_4010_9d0e_6c902a6c6ca9.slice/crio-71600684efcaa1e5844c03c15fa11843a42a08f10b013fef0a7b5029085df17d WatchSource:0}: Error finding container 71600684efcaa1e5844c03c15fa11843a42a08f10b013fef0a7b5029085df17d: Status 404 returned error can't find the container with id 71600684efcaa1e5844c03c15fa11843a42a08f10b013fef0a7b5029085df17d Jul 19 08:30:07.907722 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:07.907680 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 19 08:30:07.907824 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:07.907795 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 19 08:30:07.907865 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:07.907837 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 19 08:30:08.036994 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:08.036909 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:30:08.339847 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:08.339752 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-49f9c5-openshift-default-777b858b69-gzbsz" event={"ID":"1cf877b7-40dd-4010-9d0e-6c902a6c6ca9","Type":"ContainerStarted","Data":"fa6797557f9a729c5633a6bbbcb7dd0ff91992c97324aef7cc0a9fcfe66587c9"} Jul 19 08:30:08.339847 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:08.339818 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-49f9c5-openshift-default-777b858b69-gzbsz" event={"ID":"1cf877b7-40dd-4010-9d0e-6c902a6c6ca9","Type":"ContainerStarted","Data":"71600684efcaa1e5844c03c15fa11843a42a08f10b013fef0a7b5029085df17d"} Jul 19 08:30:08.360572 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:08.360488 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-49f9c5-openshift-default-777b858b69-gzbsz" podStartSLOduration=1.360467653 podStartE2EDuration="1.360467653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 08:30:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:30:08.358842462 +0000 UTC m=+2874.713152123" watchObservedRunningTime="2026-07-19 08:30:08.360467653 +0000 UTC m=+2874.714777238" Jul 19 08:30:08.758158 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:08.758115 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-49f9c5-openshift-default-777b858b69-gzbsz" Jul 19 08:30:08.763572 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:08.763539 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-49f9c5-openshift-default-777b858b69-gzbsz" Jul 19 08:30:09.345606 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:09.345566 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-49f9c5-openshift-default-777b858b69-gzbsz" Jul 19 08:30:09.346639 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:09.346619 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-49f9c5-openshift-default-777b858b69-gzbsz" Jul 19 08:30:10.240823 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:10.240779 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:30:10.341048 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:10.341005 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:30:10.441265 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:10.441219 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-49f9c5-8478896568-j6226"] Jul 19 08:30:10.451891 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:10.451858 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49f9c5-8478896568-j6226" Jul 19 08:30:10.454910 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:10.454875 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-49f9c5-8478896568-j6226"] Jul 19 08:30:10.455991 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:10.455954 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-49f9c5-serving-cert\"" Jul 19 08:30:10.456580 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:10.456274 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-tdbkh\"" Jul 19 08:30:10.457615 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:10.457060 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 19 08:30:10.458018 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:10.457996 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 19 08:30:10.476132 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:10.476102 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:30:10.511438 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:10.511330 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c200891b-eb65-4c75-b356-46434d934769-openshift-service-ca\") pod \"maas-api-e2e-inf-a-49f9c5-8478896568-j6226\" (UID: \"c200891b-eb65-4c75-b356-46434d934769\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49f9c5-8478896568-j6226" Jul 19 08:30:10.511438 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:10.511382 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c200891b-eb65-4c75-b356-46434d934769-maas-api-tls\") pod \"maas-api-e2e-inf-a-49f9c5-8478896568-j6226\" (UID: \"c200891b-eb65-4c75-b356-46434d934769\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49f9c5-8478896568-j6226" Jul 19 08:30:10.511733 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:10.511497 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cnwr\" (UniqueName: \"kubernetes.io/projected/c200891b-eb65-4c75-b356-46434d934769-kube-api-access-7cnwr\") pod \"maas-api-e2e-inf-a-49f9c5-8478896568-j6226\" (UID: \"c200891b-eb65-4c75-b356-46434d934769\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49f9c5-8478896568-j6226" Jul 19 08:30:10.612599 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:10.612559 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cnwr\" (UniqueName: \"kubernetes.io/projected/c200891b-eb65-4c75-b356-46434d934769-kube-api-access-7cnwr\") pod \"maas-api-e2e-inf-a-49f9c5-8478896568-j6226\" (UID: \"c200891b-eb65-4c75-b356-46434d934769\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49f9c5-8478896568-j6226" Jul 19 08:30:10.612821 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:10.612636 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c200891b-eb65-4c75-b356-46434d934769-openshift-service-ca\") pod \"maas-api-e2e-inf-a-49f9c5-8478896568-j6226\" (UID: \"c200891b-eb65-4c75-b356-46434d934769\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49f9c5-8478896568-j6226" Jul 19 08:30:10.612821 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:10.612664 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c200891b-eb65-4c75-b356-46434d934769-maas-api-tls\") pod \"maas-api-e2e-inf-a-49f9c5-8478896568-j6226\" (UID: \"c200891b-eb65-4c75-b356-46434d934769\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49f9c5-8478896568-j6226" Jul 19 08:30:10.613360 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:10.613332 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c200891b-eb65-4c75-b356-46434d934769-openshift-service-ca\") pod \"maas-api-e2e-inf-a-49f9c5-8478896568-j6226\" (UID: \"c200891b-eb65-4c75-b356-46434d934769\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49f9c5-8478896568-j6226" Jul 19 08:30:10.615248 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:10.615220 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c200891b-eb65-4c75-b356-46434d934769-maas-api-tls\") pod \"maas-api-e2e-inf-a-49f9c5-8478896568-j6226\" (UID: \"c200891b-eb65-4c75-b356-46434d934769\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49f9c5-8478896568-j6226" Jul 19 08:30:10.620598 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:10.620569 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cnwr\" (UniqueName: \"kubernetes.io/projected/c200891b-eb65-4c75-b356-46434d934769-kube-api-access-7cnwr\") pod \"maas-api-e2e-inf-a-49f9c5-8478896568-j6226\" (UID: \"c200891b-eb65-4c75-b356-46434d934769\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49f9c5-8478896568-j6226" Jul 19 08:30:10.642644 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:10.642607 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:30:10.768247 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:10.768148 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49f9c5-8478896568-j6226" Jul 19 08:30:10.913698 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:10.913636 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-49f9c5-8478896568-j6226"] Jul 19 08:30:10.916067 ip-10-0-136-70 kubenswrapper[2574]: W0719 08:30:10.916031 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc200891b_eb65_4c75_b356_46434d934769.slice/crio-12935a4fa4a070cb2ca63287415a00a04e68e3d33684692d63ae2b14335153e6 WatchSource:0}: Error finding container 12935a4fa4a070cb2ca63287415a00a04e68e3d33684692d63ae2b14335153e6: Status 404 returned error can't find the container with id 12935a4fa4a070cb2ca63287415a00a04e68e3d33684692d63ae2b14335153e6 Jul 19 08:30:11.011755 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:11.011710 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:30:11.112642 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:11.112539 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:30:11.356580 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:11.356492 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49f9c5-8478896568-j6226" event={"ID":"c200891b-eb65-4c75-b356-46434d934769","Type":"ContainerStarted","Data":"12935a4fa4a070cb2ca63287415a00a04e68e3d33684692d63ae2b14335153e6"} Jul 19 08:30:12.363946 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:12.363911 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49f9c5-8478896568-j6226" event={"ID":"c200891b-eb65-4c75-b356-46434d934769","Type":"ContainerStarted","Data":"8f591cb20ad7df4c06ed979d592819680cee913c423e66accea23b962c749379"} Jul 19 08:30:12.364374 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:12.364042 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49f9c5-8478896568-j6226" Jul 19 08:30:12.383149 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:12.383100 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49f9c5-8478896568-j6226" podStartSLOduration=1.854730176 podStartE2EDuration="2.383083527s" totalImagesPullingTime="528.353351ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 08:30:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:30:12.381659432 +0000 UTC m=+2878.735969033" watchObservedRunningTime="2026-07-19 08:30:12.383083527 +0000 UTC m=+2878.737393127" Jul 19 08:30:14.755093 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:14.755054 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:30:15.068383 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:15.068294 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:30:15.339320 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:15.339216 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-916a44-openshift-default-56bfbfdfbf-v5mkx"] Jul 19 08:30:15.345604 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:15.345573 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-916a44-openshift-default-56bfbfdfbf-v5mkx" Jul 19 08:30:15.348400 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:15.348373 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-916a44-openshift-default-dockercfg-fmqnl\"" Jul 19 08:30:15.353596 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:15.353568 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-916a44-openshift-default-56bfbfdfbf-v5mkx"] Jul 19 08:30:15.419183 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:15.419146 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:30:15.455356 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:15.455323 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e1d475b-2f4c-4224-8ec9-ebc22d8fe366-credential-socket\") pod \"e2e-inf-b-916a44-openshift-default-56bfbfdfbf-v5mkx\" (UID: \"6e1d475b-2f4c-4224-8ec9-ebc22d8fe366\") " pod="openshift-ingress/e2e-inf-b-916a44-openshift-default-56bfbfdfbf-v5mkx" Jul 19 08:30:15.455356 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:15.455355 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e1d475b-2f4c-4224-8ec9-ebc22d8fe366-workload-certs\") pod \"e2e-inf-b-916a44-openshift-default-56bfbfdfbf-v5mkx\" (UID: \"6e1d475b-2f4c-4224-8ec9-ebc22d8fe366\") " pod="openshift-ingress/e2e-inf-b-916a44-openshift-default-56bfbfdfbf-v5mkx" Jul 19 08:30:15.455644 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:15.455381 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e1d475b-2f4c-4224-8ec9-ebc22d8fe366-istio-podinfo\") pod \"e2e-inf-b-916a44-openshift-default-56bfbfdfbf-v5mkx\" (UID: \"6e1d475b-2f4c-4224-8ec9-ebc22d8fe366\") " pod="openshift-ingress/e2e-inf-b-916a44-openshift-default-56bfbfdfbf-v5mkx" Jul 19 08:30:15.455644 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:15.455441 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e1d475b-2f4c-4224-8ec9-ebc22d8fe366-workload-socket\") pod \"e2e-inf-b-916a44-openshift-default-56bfbfdfbf-v5mkx\" (UID: \"6e1d475b-2f4c-4224-8ec9-ebc22d8fe366\") " pod="openshift-ingress/e2e-inf-b-916a44-openshift-default-56bfbfdfbf-v5mkx" Jul 19 08:30:15.455644 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:15.455603 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e1d475b-2f4c-4224-8ec9-ebc22d8fe366-istio-data\") pod \"e2e-inf-b-916a44-openshift-default-56bfbfdfbf-v5mkx\" (UID: \"6e1d475b-2f4c-4224-8ec9-ebc22d8fe366\") " pod="openshift-ingress/e2e-inf-b-916a44-openshift-default-56bfbfdfbf-v5mkx" Jul 19 08:30:15.455807 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:15.455670 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e1d475b-2f4c-4224-8ec9-ebc22d8fe366-istio-envoy\") pod \"e2e-inf-b-916a44-openshift-default-56bfbfdfbf-v5mkx\" (UID: \"6e1d475b-2f4c-4224-8ec9-ebc22d8fe366\") " pod="openshift-ingress/e2e-inf-b-916a44-openshift-default-56bfbfdfbf-v5mkx" Jul 19 08:30:15.455807 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:15.455702 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e1d475b-2f4c-4224-8ec9-ebc22d8fe366-istiod-ca-cert\") pod \"e2e-inf-b-916a44-openshift-default-56bfbfdfbf-v5mkx\" (UID: \"6e1d475b-2f4c-4224-8ec9-ebc22d8fe366\") " pod="openshift-ingress/e2e-inf-b-916a44-openshift-default-56bfbfdfbf-v5mkx" Jul 19 08:30:15.455807 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:15.455730 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvrbl\" (UniqueName: \"kubernetes.io/projected/6e1d475b-2f4c-4224-8ec9-ebc22d8fe366-kube-api-access-jvrbl\") pod \"e2e-inf-b-916a44-openshift-default-56bfbfdfbf-v5mkx\" (UID: \"6e1d475b-2f4c-4224-8ec9-ebc22d8fe366\") " pod="openshift-ingress/e2e-inf-b-916a44-openshift-default-56bfbfdfbf-v5mkx" Jul 19 08:30:15.455941 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:15.455824 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e1d475b-2f4c-4224-8ec9-ebc22d8fe366-istio-token\") pod \"e2e-inf-b-916a44-openshift-default-56bfbfdfbf-v5mkx\" (UID: \"6e1d475b-2f4c-4224-8ec9-ebc22d8fe366\") " pod="openshift-ingress/e2e-inf-b-916a44-openshift-default-56bfbfdfbf-v5mkx" Jul 19 08:30:15.519751 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:15.519694 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:30:15.557241 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:15.557204 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e1d475b-2f4c-4224-8ec9-ebc22d8fe366-istio-envoy\") pod \"e2e-inf-b-916a44-openshift-default-56bfbfdfbf-v5mkx\" (UID: \"6e1d475b-2f4c-4224-8ec9-ebc22d8fe366\") " pod="openshift-ingress/e2e-inf-b-916a44-openshift-default-56bfbfdfbf-v5mkx" Jul 19 08:30:15.557241 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:15.557244 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e1d475b-2f4c-4224-8ec9-ebc22d8fe366-istiod-ca-cert\") pod \"e2e-inf-b-916a44-openshift-default-56bfbfdfbf-v5mkx\" (UID: \"6e1d475b-2f4c-4224-8ec9-ebc22d8fe366\") " pod="openshift-ingress/e2e-inf-b-916a44-openshift-default-56bfbfdfbf-v5mkx" Jul 19 08:30:15.557499 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:15.557268 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvrbl\" (UniqueName: \"kubernetes.io/projected/6e1d475b-2f4c-4224-8ec9-ebc22d8fe366-kube-api-access-jvrbl\") pod \"e2e-inf-b-916a44-openshift-default-56bfbfdfbf-v5mkx\" (UID: \"6e1d475b-2f4c-4224-8ec9-ebc22d8fe366\") " pod="openshift-ingress/e2e-inf-b-916a44-openshift-default-56bfbfdfbf-v5mkx" Jul 19 08:30:15.557499 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:15.557317 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e1d475b-2f4c-4224-8ec9-ebc22d8fe366-istio-token\") pod \"e2e-inf-b-916a44-openshift-default-56bfbfdfbf-v5mkx\" (UID: \"6e1d475b-2f4c-4224-8ec9-ebc22d8fe366\") " pod="openshift-ingress/e2e-inf-b-916a44-openshift-default-56bfbfdfbf-v5mkx" Jul 19 08:30:15.557499 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:15.557365 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e1d475b-2f4c-4224-8ec9-ebc22d8fe366-credential-socket\") pod \"e2e-inf-b-916a44-openshift-default-56bfbfdfbf-v5mkx\" (UID: \"6e1d475b-2f4c-4224-8ec9-ebc22d8fe366\") " pod="openshift-ingress/e2e-inf-b-916a44-openshift-default-56bfbfdfbf-v5mkx" Jul 19 08:30:15.557499 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:15.557388 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e1d475b-2f4c-4224-8ec9-ebc22d8fe366-workload-certs\") pod \"e2e-inf-b-916a44-openshift-default-56bfbfdfbf-v5mkx\" (UID: \"6e1d475b-2f4c-4224-8ec9-ebc22d8fe366\") " pod="openshift-ingress/e2e-inf-b-916a44-openshift-default-56bfbfdfbf-v5mkx" Jul 19 08:30:15.557761 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:15.557509 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e1d475b-2f4c-4224-8ec9-ebc22d8fe366-istio-podinfo\") pod \"e2e-inf-b-916a44-openshift-default-56bfbfdfbf-v5mkx\" (UID: \"6e1d475b-2f4c-4224-8ec9-ebc22d8fe366\") " pod="openshift-ingress/e2e-inf-b-916a44-openshift-default-56bfbfdfbf-v5mkx" Jul 19 08:30:15.557761 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:15.557584 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e1d475b-2f4c-4224-8ec9-ebc22d8fe366-workload-socket\") pod \"e2e-inf-b-916a44-openshift-default-56bfbfdfbf-v5mkx\" (UID: \"6e1d475b-2f4c-4224-8ec9-ebc22d8fe366\") " pod="openshift-ingress/e2e-inf-b-916a44-openshift-default-56bfbfdfbf-v5mkx" Jul 19 08:30:15.557761 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:15.557640 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e1d475b-2f4c-4224-8ec9-ebc22d8fe366-istio-data\") pod \"e2e-inf-b-916a44-openshift-default-56bfbfdfbf-v5mkx\" (UID: \"6e1d475b-2f4c-4224-8ec9-ebc22d8fe366\") " pod="openshift-ingress/e2e-inf-b-916a44-openshift-default-56bfbfdfbf-v5mkx" Jul 19 08:30:15.557981 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:15.557762 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e1d475b-2f4c-4224-8ec9-ebc22d8fe366-workload-certs\") pod \"e2e-inf-b-916a44-openshift-default-56bfbfdfbf-v5mkx\" (UID: \"6e1d475b-2f4c-4224-8ec9-ebc22d8fe366\") " pod="openshift-ingress/e2e-inf-b-916a44-openshift-default-56bfbfdfbf-v5mkx" Jul 19 08:30:15.557981 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:15.557802 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e1d475b-2f4c-4224-8ec9-ebc22d8fe366-credential-socket\") pod \"e2e-inf-b-916a44-openshift-default-56bfbfdfbf-v5mkx\" (UID: \"6e1d475b-2f4c-4224-8ec9-ebc22d8fe366\") " pod="openshift-ingress/e2e-inf-b-916a44-openshift-default-56bfbfdfbf-v5mkx" Jul 19 08:30:15.557981 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:15.557944 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e1d475b-2f4c-4224-8ec9-ebc22d8fe366-istio-data\") pod \"e2e-inf-b-916a44-openshift-default-56bfbfdfbf-v5mkx\" (UID: \"6e1d475b-2f4c-4224-8ec9-ebc22d8fe366\") " pod="openshift-ingress/e2e-inf-b-916a44-openshift-default-56bfbfdfbf-v5mkx" Jul 19 08:30:15.558102 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:15.558044 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e1d475b-2f4c-4224-8ec9-ebc22d8fe366-workload-socket\") pod \"e2e-inf-b-916a44-openshift-default-56bfbfdfbf-v5mkx\" (UID: \"6e1d475b-2f4c-4224-8ec9-ebc22d8fe366\") " pod="openshift-ingress/e2e-inf-b-916a44-openshift-default-56bfbfdfbf-v5mkx" Jul 19 08:30:15.558102 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:15.558087 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e1d475b-2f4c-4224-8ec9-ebc22d8fe366-istiod-ca-cert\") pod \"e2e-inf-b-916a44-openshift-default-56bfbfdfbf-v5mkx\" (UID: \"6e1d475b-2f4c-4224-8ec9-ebc22d8fe366\") " pod="openshift-ingress/e2e-inf-b-916a44-openshift-default-56bfbfdfbf-v5mkx" Jul 19 08:30:15.559626 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:15.559602 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e1d475b-2f4c-4224-8ec9-ebc22d8fe366-istio-envoy\") pod \"e2e-inf-b-916a44-openshift-default-56bfbfdfbf-v5mkx\" (UID: \"6e1d475b-2f4c-4224-8ec9-ebc22d8fe366\") " pod="openshift-ingress/e2e-inf-b-916a44-openshift-default-56bfbfdfbf-v5mkx" Jul 19 08:30:15.560045 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:15.560025 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e1d475b-2f4c-4224-8ec9-ebc22d8fe366-istio-podinfo\") pod \"e2e-inf-b-916a44-openshift-default-56bfbfdfbf-v5mkx\" (UID: \"6e1d475b-2f4c-4224-8ec9-ebc22d8fe366\") " pod="openshift-ingress/e2e-inf-b-916a44-openshift-default-56bfbfdfbf-v5mkx" Jul 19 08:30:15.564841 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:15.564815 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e1d475b-2f4c-4224-8ec9-ebc22d8fe366-istio-token\") pod \"e2e-inf-b-916a44-openshift-default-56bfbfdfbf-v5mkx\" (UID: \"6e1d475b-2f4c-4224-8ec9-ebc22d8fe366\") " pod="openshift-ingress/e2e-inf-b-916a44-openshift-default-56bfbfdfbf-v5mkx" Jul 19 08:30:15.564950 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:15.564932 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvrbl\" (UniqueName: \"kubernetes.io/projected/6e1d475b-2f4c-4224-8ec9-ebc22d8fe366-kube-api-access-jvrbl\") pod \"e2e-inf-b-916a44-openshift-default-56bfbfdfbf-v5mkx\" (UID: \"6e1d475b-2f4c-4224-8ec9-ebc22d8fe366\") " pod="openshift-ingress/e2e-inf-b-916a44-openshift-default-56bfbfdfbf-v5mkx" Jul 19 08:30:15.661907 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:15.661813 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-916a44-openshift-default-56bfbfdfbf-v5mkx" Jul 19 08:30:15.809624 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:15.809594 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-916a44-openshift-default-56bfbfdfbf-v5mkx"] Jul 19 08:30:15.811435 ip-10-0-136-70 kubenswrapper[2574]: W0719 08:30:15.811407 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e1d475b_2f4c_4224_8ec9_ebc22d8fe366.slice/crio-0a8e155b3cd030ff95866ae8eb07b4c7dfd1786f0f67abcf68a9f45785a0acd0 WatchSource:0}: Error finding container 0a8e155b3cd030ff95866ae8eb07b4c7dfd1786f0f67abcf68a9f45785a0acd0: Status 404 returned error can't find the container with id 0a8e155b3cd030ff95866ae8eb07b4c7dfd1786f0f67abcf68a9f45785a0acd0 Jul 19 08:30:15.813894 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:15.813855 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 19 08:30:15.814006 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:15.813939 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 19 08:30:15.814006 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:15.813987 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 19 08:30:16.386846 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:16.386805 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-916a44-openshift-default-56bfbfdfbf-v5mkx" event={"ID":"6e1d475b-2f4c-4224-8ec9-ebc22d8fe366","Type":"ContainerStarted","Data":"70a4a174cb4c9d7dfaa923139d33103ea9b33f0398584e1dda8186a3019cfa48"} Jul 19 08:30:16.387030 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:16.386863 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-916a44-openshift-default-56bfbfdfbf-v5mkx" event={"ID":"6e1d475b-2f4c-4224-8ec9-ebc22d8fe366","Type":"ContainerStarted","Data":"0a8e155b3cd030ff95866ae8eb07b4c7dfd1786f0f67abcf68a9f45785a0acd0"} Jul 19 08:30:16.407126 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:16.407071 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-916a44-openshift-default-56bfbfdfbf-v5mkx" podStartSLOduration=1.407054693 podStartE2EDuration="1.407054693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 08:30:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:30:16.405293849 +0000 UTC m=+2882.759603450" watchObservedRunningTime="2026-07-19 08:30:16.407054693 +0000 UTC m=+2882.761364294" Jul 19 08:30:16.662239 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:16.662142 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-916a44-openshift-default-56bfbfdfbf-v5mkx" Jul 19 08:30:16.667371 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:16.667346 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-916a44-openshift-default-56bfbfdfbf-v5mkx" Jul 19 08:30:17.391758 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:17.391715 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-916a44-openshift-default-56bfbfdfbf-v5mkx" Jul 19 08:30:17.392631 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:17.392614 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-916a44-openshift-default-56bfbfdfbf-v5mkx" Jul 19 08:30:18.268487 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:18.268447 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 08:30:18.361587 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:18.361543 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:30:18.375334 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:18.375310 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49f9c5-8478896568-j6226" Jul 19 08:30:18.501067 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:18.501033 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:30:18.579818 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:18.579735 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-916a44-795dfdc5d-lxlv4"] Jul 19 08:30:18.583474 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:18.583453 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-916a44-795dfdc5d-lxlv4" Jul 19 08:30:18.588542 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:18.588487 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-916a44-serving-cert\"" Jul 19 08:30:18.596145 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:18.596124 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-916a44-795dfdc5d-lxlv4"] Jul 19 08:30:18.633910 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:18.633870 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:30:18.687799 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:18.687759 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6945c2d1-3bd0-48dc-b4ca-3dbb2e8e036f-maas-api-tls\") pod \"maas-api-e2e-inf-b-916a44-795dfdc5d-lxlv4\" (UID: \"6945c2d1-3bd0-48dc-b4ca-3dbb2e8e036f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-916a44-795dfdc5d-lxlv4" Jul 19 08:30:18.687990 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:18.687946 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdgsj\" (UniqueName: \"kubernetes.io/projected/6945c2d1-3bd0-48dc-b4ca-3dbb2e8e036f-kube-api-access-jdgsj\") pod \"maas-api-e2e-inf-b-916a44-795dfdc5d-lxlv4\" (UID: \"6945c2d1-3bd0-48dc-b4ca-3dbb2e8e036f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-916a44-795dfdc5d-lxlv4" Jul 19 08:30:18.688074 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:18.688003 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6945c2d1-3bd0-48dc-b4ca-3dbb2e8e036f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-916a44-795dfdc5d-lxlv4\" (UID: \"6945c2d1-3bd0-48dc-b4ca-3dbb2e8e036f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-916a44-795dfdc5d-lxlv4" Jul 19 08:30:18.788510 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:18.788467 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdgsj\" (UniqueName: \"kubernetes.io/projected/6945c2d1-3bd0-48dc-b4ca-3dbb2e8e036f-kube-api-access-jdgsj\") pod \"maas-api-e2e-inf-b-916a44-795dfdc5d-lxlv4\" (UID: \"6945c2d1-3bd0-48dc-b4ca-3dbb2e8e036f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-916a44-795dfdc5d-lxlv4" Jul 19 08:30:18.788733 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:18.788547 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6945c2d1-3bd0-48dc-b4ca-3dbb2e8e036f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-916a44-795dfdc5d-lxlv4\" (UID: \"6945c2d1-3bd0-48dc-b4ca-3dbb2e8e036f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-916a44-795dfdc5d-lxlv4" Jul 19 08:30:18.788733 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:18.788582 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6945c2d1-3bd0-48dc-b4ca-3dbb2e8e036f-maas-api-tls\") pod \"maas-api-e2e-inf-b-916a44-795dfdc5d-lxlv4\" (UID: \"6945c2d1-3bd0-48dc-b4ca-3dbb2e8e036f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-916a44-795dfdc5d-lxlv4" Jul 19 08:30:18.788733 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:30:18.788678 2574 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-916a44-serving-cert: secret "maas-api-e2e-inf-b-916a44-serving-cert" not found Jul 19 08:30:18.788977 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:30:18.788740 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6945c2d1-3bd0-48dc-b4ca-3dbb2e8e036f-maas-api-tls podName:6945c2d1-3bd0-48dc-b4ca-3dbb2e8e036f nodeName:}" failed. No retries permitted until 2026-07-19 08:30:19.288723474 +0000 UTC m=+2885.643033053 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6945c2d1-3bd0-48dc-b4ca-3dbb2e8e036f-maas-api-tls") pod "maas-api-e2e-inf-b-916a44-795dfdc5d-lxlv4" (UID: "6945c2d1-3bd0-48dc-b4ca-3dbb2e8e036f") : secret "maas-api-e2e-inf-b-916a44-serving-cert" not found Jul 19 08:30:18.789269 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:18.789243 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6945c2d1-3bd0-48dc-b4ca-3dbb2e8e036f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-916a44-795dfdc5d-lxlv4\" (UID: \"6945c2d1-3bd0-48dc-b4ca-3dbb2e8e036f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-916a44-795dfdc5d-lxlv4" Jul 19 08:30:18.797975 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:18.797948 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdgsj\" (UniqueName: \"kubernetes.io/projected/6945c2d1-3bd0-48dc-b4ca-3dbb2e8e036f-kube-api-access-jdgsj\") pod \"maas-api-e2e-inf-b-916a44-795dfdc5d-lxlv4\" (UID: \"6945c2d1-3bd0-48dc-b4ca-3dbb2e8e036f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-916a44-795dfdc5d-lxlv4" Jul 19 08:30:19.053986 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:19.053933 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:30:19.169329 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:19.169285 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:30:19.266011 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:19.265972 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:30:19.293254 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:19.293225 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6945c2d1-3bd0-48dc-b4ca-3dbb2e8e036f-maas-api-tls\") pod \"maas-api-e2e-inf-b-916a44-795dfdc5d-lxlv4\" (UID: \"6945c2d1-3bd0-48dc-b4ca-3dbb2e8e036f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-916a44-795dfdc5d-lxlv4" Jul 19 08:30:19.295883 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:19.295855 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6945c2d1-3bd0-48dc-b4ca-3dbb2e8e036f-maas-api-tls\") pod \"maas-api-e2e-inf-b-916a44-795dfdc5d-lxlv4\" (UID: \"6945c2d1-3bd0-48dc-b4ca-3dbb2e8e036f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-916a44-795dfdc5d-lxlv4" Jul 19 08:30:19.497249 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:19.497207 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-916a44-795dfdc5d-lxlv4" Jul 19 08:30:19.690973 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:19.690946 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-916a44-795dfdc5d-lxlv4"] Jul 19 08:30:19.693261 ip-10-0-136-70 kubenswrapper[2574]: W0719 08:30:19.693232 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6945c2d1_3bd0_48dc_b4ca_3dbb2e8e036f.slice/crio-fb1df99a1dee998b46566f736cce7ae9fe72825c9dc2066cac44027ed3b19a80 WatchSource:0}: Error finding container fb1df99a1dee998b46566f736cce7ae9fe72825c9dc2066cac44027ed3b19a80: Status 404 returned error can't find the container with id fb1df99a1dee998b46566f736cce7ae9fe72825c9dc2066cac44027ed3b19a80 Jul 19 08:30:20.410856 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:20.410815 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-916a44-795dfdc5d-lxlv4" event={"ID":"6945c2d1-3bd0-48dc-b4ca-3dbb2e8e036f","Type":"ContainerStarted","Data":"0e20daff68b9d593d66597d0ba607c80a621f4873a5ef42502328ea9b8477eae"} Jul 19 08:30:20.411047 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:20.410884 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-916a44-795dfdc5d-lxlv4" event={"ID":"6945c2d1-3bd0-48dc-b4ca-3dbb2e8e036f","Type":"ContainerStarted","Data":"fb1df99a1dee998b46566f736cce7ae9fe72825c9dc2066cac44027ed3b19a80"} Jul 19 08:30:20.411047 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:20.410993 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-916a44-795dfdc5d-lxlv4" Jul 19 08:30:20.429017 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:20.428878 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-916a44-795dfdc5d-lxlv4" podStartSLOduration=1.9455032970000001 podStartE2EDuration="2.428855911s" totalImagesPullingTime="483.352614ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 08:30:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:30:20.426452009 +0000 UTC m=+2886.780761611" watchObservedRunningTime="2026-07-19 08:30:20.428855911 +0000 UTC m=+2886.783165506" Jul 19 08:30:22.859346 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:22.859310 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:30:25.386818 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:25.386748 2574 patch_prober.go:28] interesting pod/e2e-inf-a-49f9c5-openshift-default-777b858b69-gzbsz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 19 08:30:25.387271 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:25.386909 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-49f9c5-openshift-default-777b858b69-gzbsz" podUID="1cf877b7-40dd-4010-9d0e-6c902a6c6ca9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 19 08:30:26.423146 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:26.423060 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-916a44-795dfdc5d-lxlv4" Jul 19 08:30:26.449717 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:26.449685 2574 generic.go:356] "Generic (PLEG): container finished" podID="1cf877b7-40dd-4010-9d0e-6c902a6c6ca9" containerID="fa6797557f9a729c5633a6bbbcb7dd0ff91992c97324aef7cc0a9fcfe66587c9" exitCode=0 Jul 19 08:30:26.449889 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:26.449732 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-49f9c5-openshift-default-777b858b69-gzbsz" event={"ID":"1cf877b7-40dd-4010-9d0e-6c902a6c6ca9","Type":"ContainerDied","Data":"fa6797557f9a729c5633a6bbbcb7dd0ff91992c97324aef7cc0a9fcfe66587c9"} Jul 19 08:30:26.450200 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:26.450183 2574 scope.go:117] "RemoveContainer" containerID="fa6797557f9a729c5633a6bbbcb7dd0ff91992c97324aef7cc0a9fcfe66587c9" Jul 19 08:30:26.451304 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:26.451273 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 19 08:30:26.451405 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:26.451357 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 19 08:30:26.451469 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:26.451406 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 19 08:30:27.457157 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:27.457120 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-49f9c5-openshift-default-777b858b69-gzbsz" event={"ID":"1cf877b7-40dd-4010-9d0e-6c902a6c6ca9","Type":"ContainerStarted","Data":"cb585d3875ddd0d67a1e347d1e7e463f534f0759fc191ebd22606359cd29d7aa"} Jul 19 08:30:27.757950 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:27.757911 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-49f9c5-openshift-default-777b858b69-gzbsz" Jul 19 08:30:27.758166 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:27.758127 2574 patch_prober.go:28] interesting pod/e2e-inf-a-49f9c5-openshift-default-777b858b69-gzbsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Jul 19 08:30:27.758255 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:27.758206 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-49f9c5-openshift-default-777b858b69-gzbsz" podUID="1cf877b7-40dd-4010-9d0e-6c902a6c6ca9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Jul 19 08:30:28.758098 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:28.757601 2574 patch_prober.go:28] interesting pod/e2e-inf-a-49f9c5-openshift-default-777b858b69-gzbsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Jul 19 08:30:28.758098 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:28.757687 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-49f9c5-openshift-default-777b858b69-gzbsz" podUID="1cf877b7-40dd-4010-9d0e-6c902a6c6ca9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Jul 19 08:30:29.757938 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:29.757896 2574 patch_prober.go:28] interesting pod/e2e-inf-a-49f9c5-openshift-default-777b858b69-gzbsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Jul 19 08:30:29.758263 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:29.758228 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-49f9c5-openshift-default-777b858b69-gzbsz" podUID="1cf877b7-40dd-4010-9d0e-6c902a6c6ca9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Jul 19 08:30:30.757944 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:30.757898 2574 patch_prober.go:28] interesting pod/e2e-inf-a-49f9c5-openshift-default-777b858b69-gzbsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Jul 19 08:30:30.758263 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:30.758233 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-49f9c5-openshift-default-777b858b69-gzbsz" podUID="1cf877b7-40dd-4010-9d0e-6c902a6c6ca9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Jul 19 08:30:31.761423 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:31.761382 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-49f9c5-openshift-default-777b858b69-gzbsz" Jul 19 08:30:31.761908 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:31.761761 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-49f9c5-openshift-default-777b858b69-gzbsz" Jul 19 08:30:31.762429 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:31.762413 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-49f9c5-openshift-default-777b858b69-gzbsz" Jul 19 08:30:31.843544 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:31.830341 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:30:31.928803 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:31.928749 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:30:32.157452 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:32.157358 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:30:33.273147 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:33.273106 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 08:30:33.433728 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:33.433678 2574 patch_prober.go:28] interesting pod/e2e-inf-b-916a44-openshift-default-56bfbfdfbf-v5mkx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 19 08:30:33.433925 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:33.433764 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-916a44-openshift-default-56bfbfdfbf-v5mkx" podUID="6e1d475b-2f4c-4224-8ec9-ebc22d8fe366" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 19 08:30:33.997474 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:33.997433 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:30:35.509096 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:35.509065 2574 generic.go:356] "Generic (PLEG): container finished" podID="6e1d475b-2f4c-4224-8ec9-ebc22d8fe366" containerID="70a4a174cb4c9d7dfaa923139d33103ea9b33f0398584e1dda8186a3019cfa48" exitCode=0 Jul 19 08:30:35.509575 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:35.509140 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-916a44-openshift-default-56bfbfdfbf-v5mkx" event={"ID":"6e1d475b-2f4c-4224-8ec9-ebc22d8fe366","Type":"ContainerDied","Data":"70a4a174cb4c9d7dfaa923139d33103ea9b33f0398584e1dda8186a3019cfa48"} Jul 19 08:30:35.509575 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:35.509559 2574 scope.go:117] "RemoveContainer" containerID="70a4a174cb4c9d7dfaa923139d33103ea9b33f0398584e1dda8186a3019cfa48" Jul 19 08:30:35.510607 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:35.510580 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 19 08:30:35.510681 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:35.510647 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 19 08:30:35.510723 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:35.510687 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 19 08:30:36.516766 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:36.516725 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-916a44-openshift-default-56bfbfdfbf-v5mkx" event={"ID":"6e1d475b-2f4c-4224-8ec9-ebc22d8fe366","Type":"ContainerStarted","Data":"600d91863bb7ff37d8464d5d856b0e6aeb8f420a04d10f1b68dc4df73ff717a6"} Jul 19 08:30:36.662737 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:36.662706 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-916a44-openshift-default-56bfbfdfbf-v5mkx" Jul 19 08:30:36.662924 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:36.662903 2574 patch_prober.go:28] interesting pod/e2e-inf-b-916a44-openshift-default-56bfbfdfbf-v5mkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Jul 19 08:30:36.662979 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:36.662960 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-916a44-openshift-default-56bfbfdfbf-v5mkx" podUID="6e1d475b-2f4c-4224-8ec9-ebc22d8fe366" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Jul 19 08:30:37.663317 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:37.663277 2574 patch_prober.go:28] interesting pod/e2e-inf-b-916a44-openshift-default-56bfbfdfbf-v5mkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Jul 19 08:30:37.663814 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:37.663338 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-916a44-openshift-default-56bfbfdfbf-v5mkx" podUID="6e1d475b-2f4c-4224-8ec9-ebc22d8fe366" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Jul 19 08:30:38.663222 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:38.663176 2574 patch_prober.go:28] interesting pod/e2e-inf-b-916a44-openshift-default-56bfbfdfbf-v5mkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Jul 19 08:30:38.663564 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:38.663245 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-916a44-openshift-default-56bfbfdfbf-v5mkx" podUID="6e1d475b-2f4c-4224-8ec9-ebc22d8fe366" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Jul 19 08:30:39.662262 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:39.662223 2574 patch_prober.go:28] interesting pod/e2e-inf-b-916a44-openshift-default-56bfbfdfbf-v5mkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Jul 19 08:30:39.662461 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:39.662282 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-916a44-openshift-default-56bfbfdfbf-v5mkx" podUID="6e1d475b-2f4c-4224-8ec9-ebc22d8fe366" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Jul 19 08:30:39.981727 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:39.981688 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:30:40.081805 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:40.081764 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:30:40.666355 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:40.666322 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-916a44-openshift-default-56bfbfdfbf-v5mkx" Jul 19 08:30:40.666605 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:40.666552 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-916a44-openshift-default-56bfbfdfbf-v5mkx" Jul 19 08:30:40.667375 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:40.667354 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-916a44-openshift-default-56bfbfdfbf-v5mkx" Jul 19 08:30:43.180342 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:43.180299 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:30:45.875330 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:45.875291 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 08:30:59.801968 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:30:59.801930 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:31:04.169965 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:31:04.169926 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 08:31:18.881822 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:31:18.881779 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 08:31:21.294982 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:31:21.294929 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:31:22.210098 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:31:22.210054 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:31:22.850814 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:31:22.850775 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:31:22.902180 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:31:22.902143 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:31:32.786996 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:31:32.786951 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-49f9c5-8478896568-j6226"] Jul 19 08:31:32.787476 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:31:32.787236 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49f9c5-8478896568-j6226" podUID="c200891b-eb65-4c75-b356-46434d934769" containerName="maas-api" containerID="cri-o://8f591cb20ad7df4c06ed979d592819680cee913c423e66accea23b962c749379" gracePeriod=30 Jul 19 08:31:32.808289 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:31:32.808251 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:31:33.249051 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:31:33.249022 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49f9c5-8478896568-j6226" Jul 19 08:31:33.387606 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:31:33.387502 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c200891b-eb65-4c75-b356-46434d934769-maas-api-tls\") pod \"c200891b-eb65-4c75-b356-46434d934769\" (UID: \"c200891b-eb65-4c75-b356-46434d934769\") " Jul 19 08:31:33.387793 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:31:33.387644 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7cnwr\" (UniqueName: \"kubernetes.io/projected/c200891b-eb65-4c75-b356-46434d934769-kube-api-access-7cnwr\") pod \"c200891b-eb65-4c75-b356-46434d934769\" (UID: \"c200891b-eb65-4c75-b356-46434d934769\") " Jul 19 08:31:33.387793 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:31:33.387693 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c200891b-eb65-4c75-b356-46434d934769-openshift-service-ca\") pod \"c200891b-eb65-4c75-b356-46434d934769\" (UID: \"c200891b-eb65-4c75-b356-46434d934769\") " Jul 19 08:31:33.388145 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:31:33.388116 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c200891b-eb65-4c75-b356-46434d934769-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c200891b-eb65-4c75-b356-46434d934769" (UID: "c200891b-eb65-4c75-b356-46434d934769"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 08:31:33.389763 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:31:33.389736 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c200891b-eb65-4c75-b356-46434d934769-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c200891b-eb65-4c75-b356-46434d934769" (UID: "c200891b-eb65-4c75-b356-46434d934769"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 08:31:33.389890 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:31:33.389872 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c200891b-eb65-4c75-b356-46434d934769-kube-api-access-7cnwr" (OuterVolumeSpecName: "kube-api-access-7cnwr") pod "c200891b-eb65-4c75-b356-46434d934769" (UID: "c200891b-eb65-4c75-b356-46434d934769"). InnerVolumeSpecName "kube-api-access-7cnwr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 08:31:33.488827 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:31:33.488786 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cnwr\" (UniqueName: \"kubernetes.io/projected/c200891b-eb65-4c75-b356-46434d934769-kube-api-access-7cnwr\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:31:33.488827 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:31:33.488819 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c200891b-eb65-4c75-b356-46434d934769-openshift-service-ca\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:31:33.488827 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:31:33.488830 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c200891b-eb65-4c75-b356-46434d934769-maas-api-tls\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:31:33.836180 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:31:33.836139 2574 generic.go:356] "Generic (PLEG): container finished" podID="c200891b-eb65-4c75-b356-46434d934769" containerID="8f591cb20ad7df4c06ed979d592819680cee913c423e66accea23b962c749379" exitCode=0 Jul 19 08:31:33.836710 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:31:33.836193 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49f9c5-8478896568-j6226" Jul 19 08:31:33.836710 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:31:33.836197 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49f9c5-8478896568-j6226" event={"ID":"c200891b-eb65-4c75-b356-46434d934769","Type":"ContainerDied","Data":"8f591cb20ad7df4c06ed979d592819680cee913c423e66accea23b962c749379"} Jul 19 08:31:33.836710 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:31:33.836317 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49f9c5-8478896568-j6226" event={"ID":"c200891b-eb65-4c75-b356-46434d934769","Type":"ContainerDied","Data":"12935a4fa4a070cb2ca63287415a00a04e68e3d33684692d63ae2b14335153e6"} Jul 19 08:31:33.836710 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:31:33.836339 2574 scope.go:117] "RemoveContainer" containerID="8f591cb20ad7df4c06ed979d592819680cee913c423e66accea23b962c749379" Jul 19 08:31:33.845935 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:31:33.845917 2574 scope.go:117] "RemoveContainer" containerID="8f591cb20ad7df4c06ed979d592819680cee913c423e66accea23b962c749379" Jul 19 08:31:33.846198 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:31:33.846178 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f591cb20ad7df4c06ed979d592819680cee913c423e66accea23b962c749379\": container with ID starting with 8f591cb20ad7df4c06ed979d592819680cee913c423e66accea23b962c749379 not found: ID does not exist" containerID="8f591cb20ad7df4c06ed979d592819680cee913c423e66accea23b962c749379" Jul 19 08:31:33.846266 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:31:33.846205 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f591cb20ad7df4c06ed979d592819680cee913c423e66accea23b962c749379"} err="failed to get container status \"8f591cb20ad7df4c06ed979d592819680cee913c423e66accea23b962c749379\": rpc error: code = NotFound desc = could not find container \"8f591cb20ad7df4c06ed979d592819680cee913c423e66accea23b962c749379\": container with ID starting with 8f591cb20ad7df4c06ed979d592819680cee913c423e66accea23b962c749379 not found: ID does not exist" Jul 19 08:31:33.857265 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:31:33.857241 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-49f9c5-8478896568-j6226"] Jul 19 08:31:33.861057 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:31:33.861034 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-49f9c5-8478896568-j6226"] Jul 19 08:31:34.559851 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:31:34.559815 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c200891b-eb65-4c75-b356-46434d934769" path="/var/lib/kubelet/pods/c200891b-eb65-4c75-b356-46434d934769/volumes" Jul 19 08:31:35.680485 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:31:35.680441 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 08:31:37.399532 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:31:37.399455 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:31:43.288164 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:31:43.288120 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:31:43.391873 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:31:43.391837 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:31:48.073428 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:31:48.073393 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-49f9c5-openshift-default-777b858b69-gzbsz"] Jul 19 08:31:48.073987 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:31:48.073628 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-49f9c5-openshift-default-777b858b69-gzbsz" podUID="1cf877b7-40dd-4010-9d0e-6c902a6c6ca9" containerName="istio-proxy" containerID="cri-o://cb585d3875ddd0d67a1e347d1e7e463f534f0759fc191ebd22606359cd29d7aa" gracePeriod=30 Jul 19 08:31:48.097727 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:31:48.097691 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:31:48.328169 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:31:48.328068 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:31:49.042640 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:31:49.042601 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:31:49.143536 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:31:49.143475 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:31:53.334581 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:31:53.334553 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-49f9c5-openshift-default-777b858b69-gzbsz" Jul 19 08:31:53.373842 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:31:53.373749 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cf877b7-40dd-4010-9d0e-6c902a6c6ca9-istio-envoy\") pod \"1cf877b7-40dd-4010-9d0e-6c902a6c6ca9\" (UID: \"1cf877b7-40dd-4010-9d0e-6c902a6c6ca9\") " Jul 19 08:31:53.373842 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:31:53.373813 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cf877b7-40dd-4010-9d0e-6c902a6c6ca9-credential-socket\") pod \"1cf877b7-40dd-4010-9d0e-6c902a6c6ca9\" (UID: \"1cf877b7-40dd-4010-9d0e-6c902a6c6ca9\") " Jul 19 08:31:53.374080 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:31:53.373882 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cf877b7-40dd-4010-9d0e-6c902a6c6ca9-istio-token\") pod \"1cf877b7-40dd-4010-9d0e-6c902a6c6ca9\" (UID: \"1cf877b7-40dd-4010-9d0e-6c902a6c6ca9\") " Jul 19 08:31:53.374080 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:31:53.373905 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cf877b7-40dd-4010-9d0e-6c902a6c6ca9-istio-podinfo\") pod \"1cf877b7-40dd-4010-9d0e-6c902a6c6ca9\" (UID: \"1cf877b7-40dd-4010-9d0e-6c902a6c6ca9\") " Jul 19 08:31:53.374080 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:31:53.373922 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cf877b7-40dd-4010-9d0e-6c902a6c6ca9-workload-socket\") pod \"1cf877b7-40dd-4010-9d0e-6c902a6c6ca9\" (UID: \"1cf877b7-40dd-4010-9d0e-6c902a6c6ca9\") " Jul 19 08:31:53.374080 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:31:53.373948 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cf877b7-40dd-4010-9d0e-6c902a6c6ca9-istiod-ca-cert\") pod \"1cf877b7-40dd-4010-9d0e-6c902a6c6ca9\" (UID: \"1cf877b7-40dd-4010-9d0e-6c902a6c6ca9\") " Jul 19 08:31:53.374080 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:31:53.373975 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5wf92\" (UniqueName: \"kubernetes.io/projected/1cf877b7-40dd-4010-9d0e-6c902a6c6ca9-kube-api-access-5wf92\") pod \"1cf877b7-40dd-4010-9d0e-6c902a6c6ca9\" (UID: \"1cf877b7-40dd-4010-9d0e-6c902a6c6ca9\") " Jul 19 08:31:53.374080 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:31:53.374004 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cf877b7-40dd-4010-9d0e-6c902a6c6ca9-workload-certs\") pod \"1cf877b7-40dd-4010-9d0e-6c902a6c6ca9\" (UID: \"1cf877b7-40dd-4010-9d0e-6c902a6c6ca9\") " Jul 19 08:31:53.374080 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:31:53.374044 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cf877b7-40dd-4010-9d0e-6c902a6c6ca9-istio-data\") pod \"1cf877b7-40dd-4010-9d0e-6c902a6c6ca9\" (UID: \"1cf877b7-40dd-4010-9d0e-6c902a6c6ca9\") " Jul 19 08:31:53.374456 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:31:53.374196 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cf877b7-40dd-4010-9d0e-6c902a6c6ca9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1cf877b7-40dd-4010-9d0e-6c902a6c6ca9" (UID: "1cf877b7-40dd-4010-9d0e-6c902a6c6ca9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 08:31:53.374456 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:31:53.374335 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cf877b7-40dd-4010-9d0e-6c902a6c6ca9-workload-socket\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:31:53.374591 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:31:53.374477 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cf877b7-40dd-4010-9d0e-6c902a6c6ca9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1cf877b7-40dd-4010-9d0e-6c902a6c6ca9" (UID: "1cf877b7-40dd-4010-9d0e-6c902a6c6ca9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 08:31:53.374822 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:31:53.374686 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cf877b7-40dd-4010-9d0e-6c902a6c6ca9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1cf877b7-40dd-4010-9d0e-6c902a6c6ca9" (UID: "1cf877b7-40dd-4010-9d0e-6c902a6c6ca9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 08:31:53.374822 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:31:53.374704 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cf877b7-40dd-4010-9d0e-6c902a6c6ca9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1cf877b7-40dd-4010-9d0e-6c902a6c6ca9" (UID: "1cf877b7-40dd-4010-9d0e-6c902a6c6ca9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 08:31:53.375111 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:31:53.375046 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cf877b7-40dd-4010-9d0e-6c902a6c6ca9-istio-data" (OuterVolumeSpecName: "istio-data") pod "1cf877b7-40dd-4010-9d0e-6c902a6c6ca9" (UID: "1cf877b7-40dd-4010-9d0e-6c902a6c6ca9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 08:31:53.376200 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:31:53.376174 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cf877b7-40dd-4010-9d0e-6c902a6c6ca9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1cf877b7-40dd-4010-9d0e-6c902a6c6ca9" (UID: "1cf877b7-40dd-4010-9d0e-6c902a6c6ca9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 08:31:53.376471 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:31:53.376452 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1cf877b7-40dd-4010-9d0e-6c902a6c6ca9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1cf877b7-40dd-4010-9d0e-6c902a6c6ca9" (UID: "1cf877b7-40dd-4010-9d0e-6c902a6c6ca9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 19 08:31:53.376749 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:31:53.376707 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cf877b7-40dd-4010-9d0e-6c902a6c6ca9-istio-token" (OuterVolumeSpecName: "istio-token") pod "1cf877b7-40dd-4010-9d0e-6c902a6c6ca9" (UID: "1cf877b7-40dd-4010-9d0e-6c902a6c6ca9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 08:31:53.376843 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:31:53.376794 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cf877b7-40dd-4010-9d0e-6c902a6c6ca9-kube-api-access-5wf92" (OuterVolumeSpecName: "kube-api-access-5wf92") pod "1cf877b7-40dd-4010-9d0e-6c902a6c6ca9" (UID: "1cf877b7-40dd-4010-9d0e-6c902a6c6ca9"). InnerVolumeSpecName "kube-api-access-5wf92". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 08:31:53.475321 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:31:53.475282 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cf877b7-40dd-4010-9d0e-6c902a6c6ca9-istio-envoy\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:31:53.475321 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:31:53.475314 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cf877b7-40dd-4010-9d0e-6c902a6c6ca9-credential-socket\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:31:53.475321 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:31:53.475323 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cf877b7-40dd-4010-9d0e-6c902a6c6ca9-istio-token\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:31:53.475321 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:31:53.475331 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cf877b7-40dd-4010-9d0e-6c902a6c6ca9-istio-podinfo\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:31:53.475649 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:31:53.475339 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cf877b7-40dd-4010-9d0e-6c902a6c6ca9-istiod-ca-cert\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:31:53.475649 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:31:53.475347 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wf92\" (UniqueName: \"kubernetes.io/projected/1cf877b7-40dd-4010-9d0e-6c902a6c6ca9-kube-api-access-5wf92\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:31:53.475649 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:31:53.475355 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cf877b7-40dd-4010-9d0e-6c902a6c6ca9-workload-certs\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:31:53.475649 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:31:53.475363 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cf877b7-40dd-4010-9d0e-6c902a6c6ca9-istio-data\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:31:53.940605 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:31:53.940568 2574 generic.go:356] "Generic (PLEG): container finished" podID="1cf877b7-40dd-4010-9d0e-6c902a6c6ca9" containerID="cb585d3875ddd0d67a1e347d1e7e463f534f0759fc191ebd22606359cd29d7aa" exitCode=0 Jul 19 08:31:53.940828 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:31:53.940615 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-49f9c5-openshift-default-777b858b69-gzbsz" Jul 19 08:31:53.940828 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:31:53.940660 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-49f9c5-openshift-default-777b858b69-gzbsz" event={"ID":"1cf877b7-40dd-4010-9d0e-6c902a6c6ca9","Type":"ContainerDied","Data":"cb585d3875ddd0d67a1e347d1e7e463f534f0759fc191ebd22606359cd29d7aa"} Jul 19 08:31:53.940828 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:31:53.940721 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-49f9c5-openshift-default-777b858b69-gzbsz" event={"ID":"1cf877b7-40dd-4010-9d0e-6c902a6c6ca9","Type":"ContainerDied","Data":"71600684efcaa1e5844c03c15fa11843a42a08f10b013fef0a7b5029085df17d"} Jul 19 08:31:53.940828 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:31:53.940752 2574 scope.go:117] "RemoveContainer" containerID="cb585d3875ddd0d67a1e347d1e7e463f534f0759fc191ebd22606359cd29d7aa" Jul 19 08:31:53.953660 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:31:53.953636 2574 scope.go:117] "RemoveContainer" containerID="fa6797557f9a729c5633a6bbbcb7dd0ff91992c97324aef7cc0a9fcfe66587c9" Jul 19 08:31:53.962237 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:31:53.962217 2574 scope.go:117] "RemoveContainer" containerID="cb585d3875ddd0d67a1e347d1e7e463f534f0759fc191ebd22606359cd29d7aa" Jul 19 08:31:53.962550 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:31:53.962506 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb585d3875ddd0d67a1e347d1e7e463f534f0759fc191ebd22606359cd29d7aa\": container with ID starting with cb585d3875ddd0d67a1e347d1e7e463f534f0759fc191ebd22606359cd29d7aa not found: ID does not exist" containerID="cb585d3875ddd0d67a1e347d1e7e463f534f0759fc191ebd22606359cd29d7aa" Jul 19 08:31:53.962657 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:31:53.962560 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb585d3875ddd0d67a1e347d1e7e463f534f0759fc191ebd22606359cd29d7aa"} err="failed to get container status \"cb585d3875ddd0d67a1e347d1e7e463f534f0759fc191ebd22606359cd29d7aa\": rpc error: code = NotFound desc = could not find container \"cb585d3875ddd0d67a1e347d1e7e463f534f0759fc191ebd22606359cd29d7aa\": container with ID starting with cb585d3875ddd0d67a1e347d1e7e463f534f0759fc191ebd22606359cd29d7aa not found: ID does not exist" Jul 19 08:31:53.962657 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:31:53.962587 2574 scope.go:117] "RemoveContainer" containerID="fa6797557f9a729c5633a6bbbcb7dd0ff91992c97324aef7cc0a9fcfe66587c9" Jul 19 08:31:53.962921 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:31:53.962901 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa6797557f9a729c5633a6bbbcb7dd0ff91992c97324aef7cc0a9fcfe66587c9\": container with ID starting with fa6797557f9a729c5633a6bbbcb7dd0ff91992c97324aef7cc0a9fcfe66587c9 not found: ID does not exist" containerID="fa6797557f9a729c5633a6bbbcb7dd0ff91992c97324aef7cc0a9fcfe66587c9" Jul 19 08:31:53.962994 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:31:53.962935 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa6797557f9a729c5633a6bbbcb7dd0ff91992c97324aef7cc0a9fcfe66587c9"} err="failed to get container status \"fa6797557f9a729c5633a6bbbcb7dd0ff91992c97324aef7cc0a9fcfe66587c9\": rpc error: code = NotFound desc = could not find container \"fa6797557f9a729c5633a6bbbcb7dd0ff91992c97324aef7cc0a9fcfe66587c9\": container with ID starting with fa6797557f9a729c5633a6bbbcb7dd0ff91992c97324aef7cc0a9fcfe66587c9 not found: ID does not exist" Jul 19 08:31:53.965785 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:31:53.965761 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-49f9c5-openshift-default-777b858b69-gzbsz"] Jul 19 08:31:53.969288 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:31:53.969267 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-49f9c5-openshift-default-777b858b69-gzbsz"] Jul 19 08:31:54.559243 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:31:54.559205 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1cf877b7-40dd-4010-9d0e-6c902a6c6ca9" path="/var/lib/kubelet/pods/1cf877b7-40dd-4010-9d0e-6c902a6c6ca9/volumes" Jul 19 08:31:58.869156 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:31:58.869091 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-916a44-795dfdc5d-lxlv4"] Jul 19 08:31:58.869898 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:31:58.869335 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-916a44-795dfdc5d-lxlv4" podUID="6945c2d1-3bd0-48dc-b4ca-3dbb2e8e036f" containerName="maas-api" containerID="cri-o://0e20daff68b9d593d66597d0ba607c80a621f4873a5ef42502328ea9b8477eae" gracePeriod=30 Jul 19 08:31:59.051112 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:31:59.051051 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:32:00.127998 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:00.127970 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-916a44-795dfdc5d-lxlv4" Jul 19 08:32:00.232220 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:00.232182 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jdgsj\" (UniqueName: \"kubernetes.io/projected/6945c2d1-3bd0-48dc-b4ca-3dbb2e8e036f-kube-api-access-jdgsj\") pod \"6945c2d1-3bd0-48dc-b4ca-3dbb2e8e036f\" (UID: \"6945c2d1-3bd0-48dc-b4ca-3dbb2e8e036f\") " Jul 19 08:32:00.232445 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:00.232252 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6945c2d1-3bd0-48dc-b4ca-3dbb2e8e036f-maas-api-tls\") pod \"6945c2d1-3bd0-48dc-b4ca-3dbb2e8e036f\" (UID: \"6945c2d1-3bd0-48dc-b4ca-3dbb2e8e036f\") " Jul 19 08:32:00.232445 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:00.232292 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6945c2d1-3bd0-48dc-b4ca-3dbb2e8e036f-openshift-service-ca\") pod \"6945c2d1-3bd0-48dc-b4ca-3dbb2e8e036f\" (UID: \"6945c2d1-3bd0-48dc-b4ca-3dbb2e8e036f\") " Jul 19 08:32:00.232745 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:00.232712 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6945c2d1-3bd0-48dc-b4ca-3dbb2e8e036f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6945c2d1-3bd0-48dc-b4ca-3dbb2e8e036f" (UID: "6945c2d1-3bd0-48dc-b4ca-3dbb2e8e036f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 08:32:00.234626 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:00.234594 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6945c2d1-3bd0-48dc-b4ca-3dbb2e8e036f-kube-api-access-jdgsj" (OuterVolumeSpecName: "kube-api-access-jdgsj") pod "6945c2d1-3bd0-48dc-b4ca-3dbb2e8e036f" (UID: "6945c2d1-3bd0-48dc-b4ca-3dbb2e8e036f"). InnerVolumeSpecName "kube-api-access-jdgsj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 08:32:00.234626 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:00.234607 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6945c2d1-3bd0-48dc-b4ca-3dbb2e8e036f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6945c2d1-3bd0-48dc-b4ca-3dbb2e8e036f" (UID: "6945c2d1-3bd0-48dc-b4ca-3dbb2e8e036f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 08:32:00.333426 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:00.333388 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6945c2d1-3bd0-48dc-b4ca-3dbb2e8e036f-maas-api-tls\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:32:00.333426 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:00.333420 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6945c2d1-3bd0-48dc-b4ca-3dbb2e8e036f-openshift-service-ca\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:32:00.333426 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:00.333430 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdgsj\" (UniqueName: \"kubernetes.io/projected/6945c2d1-3bd0-48dc-b4ca-3dbb2e8e036f-kube-api-access-jdgsj\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:32:00.978491 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:00.978455 2574 generic.go:356] "Generic (PLEG): container finished" podID="6945c2d1-3bd0-48dc-b4ca-3dbb2e8e036f" containerID="0e20daff68b9d593d66597d0ba607c80a621f4873a5ef42502328ea9b8477eae" exitCode=0 Jul 19 08:32:00.978699 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:00.978497 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-916a44-795dfdc5d-lxlv4" Jul 19 08:32:00.978699 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:00.978547 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-916a44-795dfdc5d-lxlv4" event={"ID":"6945c2d1-3bd0-48dc-b4ca-3dbb2e8e036f","Type":"ContainerDied","Data":"0e20daff68b9d593d66597d0ba607c80a621f4873a5ef42502328ea9b8477eae"} Jul 19 08:32:00.978699 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:00.978601 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-916a44-795dfdc5d-lxlv4" event={"ID":"6945c2d1-3bd0-48dc-b4ca-3dbb2e8e036f","Type":"ContainerDied","Data":"fb1df99a1dee998b46566f736cce7ae9fe72825c9dc2066cac44027ed3b19a80"} Jul 19 08:32:00.978699 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:00.978624 2574 scope.go:117] "RemoveContainer" containerID="0e20daff68b9d593d66597d0ba607c80a621f4873a5ef42502328ea9b8477eae" Jul 19 08:32:00.987747 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:00.987728 2574 scope.go:117] "RemoveContainer" containerID="0e20daff68b9d593d66597d0ba607c80a621f4873a5ef42502328ea9b8477eae" Jul 19 08:32:00.988046 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:32:00.988027 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e20daff68b9d593d66597d0ba607c80a621f4873a5ef42502328ea9b8477eae\": container with ID starting with 0e20daff68b9d593d66597d0ba607c80a621f4873a5ef42502328ea9b8477eae not found: ID does not exist" containerID="0e20daff68b9d593d66597d0ba607c80a621f4873a5ef42502328ea9b8477eae" Jul 19 08:32:00.988092 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:00.988055 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e20daff68b9d593d66597d0ba607c80a621f4873a5ef42502328ea9b8477eae"} err="failed to get container status \"0e20daff68b9d593d66597d0ba607c80a621f4873a5ef42502328ea9b8477eae\": rpc error: code = NotFound desc = could not find container \"0e20daff68b9d593d66597d0ba607c80a621f4873a5ef42502328ea9b8477eae\": container with ID starting with 0e20daff68b9d593d66597d0ba607c80a621f4873a5ef42502328ea9b8477eae not found: ID does not exist" Jul 19 08:32:00.994808 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:00.994781 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-916a44-795dfdc5d-lxlv4"] Jul 19 08:32:01.000421 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:01.000389 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-916a44-795dfdc5d-lxlv4"] Jul 19 08:32:02.079022 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:02.078982 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 08:32:02.559074 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:02.559036 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6945c2d1-3bd0-48dc-b4ca-3dbb2e8e036f" path="/var/lib/kubelet/pods/6945c2d1-3bd0-48dc-b4ca-3dbb2e8e036f/volumes" Jul 19 08:32:03.594986 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:03.594947 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:32:09.371189 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:09.371140 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:32:09.472255 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:09.472212 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:32:14.187545 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:14.187492 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-916a44-openshift-default-56bfbfdfbf-v5mkx"] Jul 19 08:32:14.187956 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:14.187716 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-916a44-openshift-default-56bfbfdfbf-v5mkx" podUID="6e1d475b-2f4c-4224-8ec9-ebc22d8fe366" containerName="istio-proxy" containerID="cri-o://600d91863bb7ff37d8464d5d856b0e6aeb8f420a04d10f1b68dc4df73ff717a6" gracePeriod=30 Jul 19 08:32:14.226392 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:14.226341 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:32:14.441638 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:14.441544 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:32:14.769304 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:14.769268 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wmhfd_dd68d31d-f763-433a-867b-65b12eabac76/console-operator/1.log" Jul 19 08:32:14.772849 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:14.772823 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:32:14.779490 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:14.779470 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wmhfd_dd68d31d-f763-433a-867b-65b12eabac76/console-operator/1.log" Jul 19 08:32:15.073946 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:15.073816 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-5da88c-openshift-default-5c89fb85d5-s9flw"] Jul 19 08:32:15.074408 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:15.074379 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c200891b-eb65-4c75-b356-46434d934769" containerName="maas-api" Jul 19 08:32:15.074408 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:15.074410 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c200891b-eb65-4c75-b356-46434d934769" containerName="maas-api" Jul 19 08:32:15.074624 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:15.074437 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6945c2d1-3bd0-48dc-b4ca-3dbb2e8e036f" containerName="maas-api" Jul 19 08:32:15.074624 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:15.074446 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="6945c2d1-3bd0-48dc-b4ca-3dbb2e8e036f" containerName="maas-api" Jul 19 08:32:15.074624 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:15.074457 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1cf877b7-40dd-4010-9d0e-6c902a6c6ca9" containerName="istio-proxy" Jul 19 08:32:15.074624 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:15.074463 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="1cf877b7-40dd-4010-9d0e-6c902a6c6ca9" containerName="istio-proxy" Jul 19 08:32:15.074624 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:15.074590 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c200891b-eb65-4c75-b356-46434d934769" containerName="maas-api" Jul 19 08:32:15.074624 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:15.074607 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="6945c2d1-3bd0-48dc-b4ca-3dbb2e8e036f" containerName="maas-api" Jul 19 08:32:15.074624 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:15.074617 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="1cf877b7-40dd-4010-9d0e-6c902a6c6ca9" containerName="istio-proxy" Jul 19 08:32:15.074624 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:15.074626 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="1cf877b7-40dd-4010-9d0e-6c902a6c6ca9" containerName="istio-proxy" Jul 19 08:32:15.076284 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:15.076253 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:32:15.079625 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:15.079606 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5da88c-openshift-default-5c89fb85d5-s9flw" Jul 19 08:32:15.082221 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:15.082199 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-5da88c-openshift-default-dockercfg-8g5c9\"" Jul 19 08:32:15.088196 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:15.088174 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-5da88c-openshift-default-5c89fb85d5-s9flw"] Jul 19 08:32:15.171238 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:15.171199 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhv2t\" (UniqueName: \"kubernetes.io/projected/310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f-kube-api-access-hhv2t\") pod \"e2e-api-a-5da88c-openshift-default-5c89fb85d5-s9flw\" (UID: \"310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f\") " pod="openshift-ingress/e2e-api-a-5da88c-openshift-default-5c89fb85d5-s9flw" Jul 19 08:32:15.171238 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:15.171248 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f-istio-data\") pod \"e2e-api-a-5da88c-openshift-default-5c89fb85d5-s9flw\" (UID: \"310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f\") " pod="openshift-ingress/e2e-api-a-5da88c-openshift-default-5c89fb85d5-s9flw" Jul 19 08:32:15.171512 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:15.171274 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f-istio-token\") pod \"e2e-api-a-5da88c-openshift-default-5c89fb85d5-s9flw\" (UID: \"310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f\") " pod="openshift-ingress/e2e-api-a-5da88c-openshift-default-5c89fb85d5-s9flw" Jul 19 08:32:15.171512 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:15.171369 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f-istio-podinfo\") pod \"e2e-api-a-5da88c-openshift-default-5c89fb85d5-s9flw\" (UID: \"310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f\") " pod="openshift-ingress/e2e-api-a-5da88c-openshift-default-5c89fb85d5-s9flw" Jul 19 08:32:15.171512 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:15.171394 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f-credential-socket\") pod \"e2e-api-a-5da88c-openshift-default-5c89fb85d5-s9flw\" (UID: \"310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f\") " pod="openshift-ingress/e2e-api-a-5da88c-openshift-default-5c89fb85d5-s9flw" Jul 19 08:32:15.171512 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:15.171419 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f-workload-certs\") pod \"e2e-api-a-5da88c-openshift-default-5c89fb85d5-s9flw\" (UID: \"310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f\") " pod="openshift-ingress/e2e-api-a-5da88c-openshift-default-5c89fb85d5-s9flw" Jul 19 08:32:15.171709 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:15.171506 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f-istiod-ca-cert\") pod \"e2e-api-a-5da88c-openshift-default-5c89fb85d5-s9flw\" (UID: \"310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f\") " pod="openshift-ingress/e2e-api-a-5da88c-openshift-default-5c89fb85d5-s9flw" Jul 19 08:32:15.171709 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:15.171608 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f-workload-socket\") pod \"e2e-api-a-5da88c-openshift-default-5c89fb85d5-s9flw\" (UID: \"310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f\") " pod="openshift-ingress/e2e-api-a-5da88c-openshift-default-5c89fb85d5-s9flw" Jul 19 08:32:15.171709 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:15.171653 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f-istio-envoy\") pod \"e2e-api-a-5da88c-openshift-default-5c89fb85d5-s9flw\" (UID: \"310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f\") " pod="openshift-ingress/e2e-api-a-5da88c-openshift-default-5c89fb85d5-s9flw" Jul 19 08:32:15.176564 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:15.176504 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:32:15.272643 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:15.272613 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f-workload-socket\") pod \"e2e-api-a-5da88c-openshift-default-5c89fb85d5-s9flw\" (UID: \"310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f\") " pod="openshift-ingress/e2e-api-a-5da88c-openshift-default-5c89fb85d5-s9flw" Jul 19 08:32:15.272643 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:15.272646 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f-istio-envoy\") pod \"e2e-api-a-5da88c-openshift-default-5c89fb85d5-s9flw\" (UID: \"310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f\") " pod="openshift-ingress/e2e-api-a-5da88c-openshift-default-5c89fb85d5-s9flw" Jul 19 08:32:15.273215 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:15.272672 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhv2t\" (UniqueName: \"kubernetes.io/projected/310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f-kube-api-access-hhv2t\") pod \"e2e-api-a-5da88c-openshift-default-5c89fb85d5-s9flw\" (UID: \"310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f\") " pod="openshift-ingress/e2e-api-a-5da88c-openshift-default-5c89fb85d5-s9flw" Jul 19 08:32:15.273215 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:15.272691 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f-istio-data\") pod \"e2e-api-a-5da88c-openshift-default-5c89fb85d5-s9flw\" (UID: \"310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f\") " pod="openshift-ingress/e2e-api-a-5da88c-openshift-default-5c89fb85d5-s9flw" Jul 19 08:32:15.273215 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:15.272707 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f-istio-token\") pod \"e2e-api-a-5da88c-openshift-default-5c89fb85d5-s9flw\" (UID: \"310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f\") " pod="openshift-ingress/e2e-api-a-5da88c-openshift-default-5c89fb85d5-s9flw" Jul 19 08:32:15.273215 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:15.272750 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f-istio-podinfo\") pod \"e2e-api-a-5da88c-openshift-default-5c89fb85d5-s9flw\" (UID: \"310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f\") " pod="openshift-ingress/e2e-api-a-5da88c-openshift-default-5c89fb85d5-s9flw" Jul 19 08:32:15.273215 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:15.272766 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f-credential-socket\") pod \"e2e-api-a-5da88c-openshift-default-5c89fb85d5-s9flw\" (UID: \"310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f\") " pod="openshift-ingress/e2e-api-a-5da88c-openshift-default-5c89fb85d5-s9flw" Jul 19 08:32:15.273215 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:15.272791 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f-workload-certs\") pod \"e2e-api-a-5da88c-openshift-default-5c89fb85d5-s9flw\" (UID: \"310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f\") " pod="openshift-ingress/e2e-api-a-5da88c-openshift-default-5c89fb85d5-s9flw" Jul 19 08:32:15.273215 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:15.272861 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f-istiod-ca-cert\") pod \"e2e-api-a-5da88c-openshift-default-5c89fb85d5-s9flw\" (UID: \"310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f\") " pod="openshift-ingress/e2e-api-a-5da88c-openshift-default-5c89fb85d5-s9flw" Jul 19 08:32:15.273215 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:15.273190 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f-workload-socket\") pod \"e2e-api-a-5da88c-openshift-default-5c89fb85d5-s9flw\" (UID: \"310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f\") " pod="openshift-ingress/e2e-api-a-5da88c-openshift-default-5c89fb85d5-s9flw" Jul 19 08:32:15.273657 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:15.273588 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f-workload-certs\") pod \"e2e-api-a-5da88c-openshift-default-5c89fb85d5-s9flw\" (UID: \"310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f\") " pod="openshift-ingress/e2e-api-a-5da88c-openshift-default-5c89fb85d5-s9flw" Jul 19 08:32:15.273722 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:15.273233 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f-credential-socket\") pod \"e2e-api-a-5da88c-openshift-default-5c89fb85d5-s9flw\" (UID: \"310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f\") " pod="openshift-ingress/e2e-api-a-5da88c-openshift-default-5c89fb85d5-s9flw" Jul 19 08:32:15.273938 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:15.273912 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f-istio-data\") pod \"e2e-api-a-5da88c-openshift-default-5c89fb85d5-s9flw\" (UID: \"310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f\") " pod="openshift-ingress/e2e-api-a-5da88c-openshift-default-5c89fb85d5-s9flw" Jul 19 08:32:15.274027 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:15.274005 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f-istiod-ca-cert\") pod \"e2e-api-a-5da88c-openshift-default-5c89fb85d5-s9flw\" (UID: \"310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f\") " pod="openshift-ingress/e2e-api-a-5da88c-openshift-default-5c89fb85d5-s9flw" Jul 19 08:32:15.275255 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:15.275220 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f-istio-envoy\") pod \"e2e-api-a-5da88c-openshift-default-5c89fb85d5-s9flw\" (UID: \"310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f\") " pod="openshift-ingress/e2e-api-a-5da88c-openshift-default-5c89fb85d5-s9flw" Jul 19 08:32:15.280616 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:15.278894 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f-istio-podinfo\") pod \"e2e-api-a-5da88c-openshift-default-5c89fb85d5-s9flw\" (UID: \"310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f\") " pod="openshift-ingress/e2e-api-a-5da88c-openshift-default-5c89fb85d5-s9flw" Jul 19 08:32:15.282286 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:15.282262 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f-istio-token\") pod \"e2e-api-a-5da88c-openshift-default-5c89fb85d5-s9flw\" (UID: \"310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f\") " pod="openshift-ingress/e2e-api-a-5da88c-openshift-default-5c89fb85d5-s9flw" Jul 19 08:32:15.284756 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:15.284734 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhv2t\" (UniqueName: \"kubernetes.io/projected/310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f-kube-api-access-hhv2t\") pod \"e2e-api-a-5da88c-openshift-default-5c89fb85d5-s9flw\" (UID: \"310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f\") " pod="openshift-ingress/e2e-api-a-5da88c-openshift-default-5c89fb85d5-s9flw" Jul 19 08:32:15.391971 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:15.391882 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5da88c-openshift-default-5c89fb85d5-s9flw" Jul 19 08:32:15.545872 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:15.545842 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-5da88c-openshift-default-5c89fb85d5-s9flw"] Jul 19 08:32:15.547652 ip-10-0-136-70 kubenswrapper[2574]: W0719 08:32:15.547620 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod310ae9a9_00fe_404c_a0f6_2b1e0ef14d3f.slice/crio-20f2f32f0fdd63b763508032f7dc9d61c202f9d60bc396a40c5096a7dc1b3ce5 WatchSource:0}: Error finding container 20f2f32f0fdd63b763508032f7dc9d61c202f9d60bc396a40c5096a7dc1b3ce5: Status 404 returned error can't find the container with id 20f2f32f0fdd63b763508032f7dc9d61c202f9d60bc396a40c5096a7dc1b3ce5 Jul 19 08:32:15.549928 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:15.549894 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 19 08:32:15.550021 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:15.549961 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 19 08:32:15.550021 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:15.549987 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 19 08:32:16.059182 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:16.059144 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5da88c-openshift-default-5c89fb85d5-s9flw" event={"ID":"310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f","Type":"ContainerStarted","Data":"cc1ba68e15fc8c56afbf0bcdde4e6699e5de322d39e74f49795ea38c9b4484f2"} Jul 19 08:32:16.059370 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:16.059206 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5da88c-openshift-default-5c89fb85d5-s9flw" event={"ID":"310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f","Type":"ContainerStarted","Data":"20f2f32f0fdd63b763508032f7dc9d61c202f9d60bc396a40c5096a7dc1b3ce5"} Jul 19 08:32:16.077922 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:16.077869 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-5da88c-openshift-default-5c89fb85d5-s9flw" podStartSLOduration=1.077853698 podStartE2EDuration="1.077853698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 08:32:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:32:16.075876256 +0000 UTC m=+3002.430185858" watchObservedRunningTime="2026-07-19 08:32:16.077853698 +0000 UTC m=+3002.432163362" Jul 19 08:32:16.392828 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:16.392734 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-5da88c-openshift-default-5c89fb85d5-s9flw" Jul 19 08:32:16.397982 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:16.397949 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-5da88c-openshift-default-5c89fb85d5-s9flw" Jul 19 08:32:17.065231 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:17.065204 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-5da88c-openshift-default-5c89fb85d5-s9flw" Jul 19 08:32:17.066214 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:17.066193 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-5da88c-openshift-default-5c89fb85d5-s9flw" Jul 19 08:32:18.021097 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:18.021058 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:32:18.132025 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:18.131992 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5da88c-667c58dfd6-74j86"] Jul 19 08:32:18.132405 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:18.132391 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1cf877b7-40dd-4010-9d0e-6c902a6c6ca9" containerName="istio-proxy" Jul 19 08:32:18.132405 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:18.132405 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="1cf877b7-40dd-4010-9d0e-6c902a6c6ca9" containerName="istio-proxy" Jul 19 08:32:18.135737 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:18.135712 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5da88c-667c58dfd6-74j86" Jul 19 08:32:18.139122 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:18.139091 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 19 08:32:18.139122 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:18.139122 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-tdbkh\"" Jul 19 08:32:18.139122 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:18.139131 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-5da88c-serving-cert\"" Jul 19 08:32:18.140398 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:18.140378 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 19 08:32:18.157306 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:18.157284 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5da88c-667c58dfd6-74j86"] Jul 19 08:32:18.165315 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:18.165286 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:32:18.261714 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:18.261667 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:32:18.303443 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:18.303343 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9619b8aa-97a1-4901-96db-302e4502d2e6-openshift-service-ca\") pod \"maas-api-e2e-api-a-5da88c-667c58dfd6-74j86\" (UID: \"9619b8aa-97a1-4901-96db-302e4502d2e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5da88c-667c58dfd6-74j86" Jul 19 08:32:18.303443 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:18.303384 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tv5cd\" (UniqueName: \"kubernetes.io/projected/9619b8aa-97a1-4901-96db-302e4502d2e6-kube-api-access-tv5cd\") pod \"maas-api-e2e-api-a-5da88c-667c58dfd6-74j86\" (UID: \"9619b8aa-97a1-4901-96db-302e4502d2e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5da88c-667c58dfd6-74j86" Jul 19 08:32:18.303443 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:18.303420 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9619b8aa-97a1-4901-96db-302e4502d2e6-maas-api-tls\") pod \"maas-api-e2e-api-a-5da88c-667c58dfd6-74j86\" (UID: \"9619b8aa-97a1-4901-96db-302e4502d2e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5da88c-667c58dfd6-74j86" Jul 19 08:32:18.361951 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:18.361914 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:32:18.404214 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:18.404178 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9619b8aa-97a1-4901-96db-302e4502d2e6-openshift-service-ca\") pod \"maas-api-e2e-api-a-5da88c-667c58dfd6-74j86\" (UID: \"9619b8aa-97a1-4901-96db-302e4502d2e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5da88c-667c58dfd6-74j86" Jul 19 08:32:18.404214 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:18.404215 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tv5cd\" (UniqueName: \"kubernetes.io/projected/9619b8aa-97a1-4901-96db-302e4502d2e6-kube-api-access-tv5cd\") pod \"maas-api-e2e-api-a-5da88c-667c58dfd6-74j86\" (UID: \"9619b8aa-97a1-4901-96db-302e4502d2e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5da88c-667c58dfd6-74j86" Jul 19 08:32:18.404439 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:18.404239 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9619b8aa-97a1-4901-96db-302e4502d2e6-maas-api-tls\") pod \"maas-api-e2e-api-a-5da88c-667c58dfd6-74j86\" (UID: \"9619b8aa-97a1-4901-96db-302e4502d2e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5da88c-667c58dfd6-74j86" Jul 19 08:32:18.404949 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:18.404924 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9619b8aa-97a1-4901-96db-302e4502d2e6-openshift-service-ca\") pod \"maas-api-e2e-api-a-5da88c-667c58dfd6-74j86\" (UID: \"9619b8aa-97a1-4901-96db-302e4502d2e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5da88c-667c58dfd6-74j86" Jul 19 08:32:18.406741 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:18.406720 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9619b8aa-97a1-4901-96db-302e4502d2e6-maas-api-tls\") pod \"maas-api-e2e-api-a-5da88c-667c58dfd6-74j86\" (UID: \"9619b8aa-97a1-4901-96db-302e4502d2e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5da88c-667c58dfd6-74j86" Jul 19 08:32:18.415592 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:18.415560 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tv5cd\" (UniqueName: \"kubernetes.io/projected/9619b8aa-97a1-4901-96db-302e4502d2e6-kube-api-access-tv5cd\") pod \"maas-api-e2e-api-a-5da88c-667c58dfd6-74j86\" (UID: \"9619b8aa-97a1-4901-96db-302e4502d2e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5da88c-667c58dfd6-74j86" Jul 19 08:32:18.447720 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:18.447670 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5da88c-667c58dfd6-74j86" Jul 19 08:32:18.589274 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:18.589247 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5da88c-667c58dfd6-74j86"] Jul 19 08:32:18.591083 ip-10-0-136-70 kubenswrapper[2574]: W0719 08:32:18.591053 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9619b8aa_97a1_4901_96db_302e4502d2e6.slice/crio-55f6861e4c42e604e537c86ce3bd2bce830486724063a3e835a0c2d77337cb71 WatchSource:0}: Error finding container 55f6861e4c42e604e537c86ce3bd2bce830486724063a3e835a0c2d77337cb71: Status 404 returned error can't find the container with id 55f6861e4c42e604e537c86ce3bd2bce830486724063a3e835a0c2d77337cb71 Jul 19 08:32:18.592535 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:18.592498 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 19 08:32:18.789999 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:18.789953 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:32:18.890025 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:18.889925 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:32:19.078682 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:19.078644 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5da88c-667c58dfd6-74j86" event={"ID":"9619b8aa-97a1-4901-96db-302e4502d2e6","Type":"ContainerStarted","Data":"55f6861e4c42e604e537c86ce3bd2bce830486724063a3e835a0c2d77337cb71"} Jul 19 08:32:19.449820 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:19.449794 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-916a44-openshift-default-56bfbfdfbf-v5mkx" Jul 19 08:32:19.614701 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:19.614662 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e1d475b-2f4c-4224-8ec9-ebc22d8fe366-workload-socket\") pod \"6e1d475b-2f4c-4224-8ec9-ebc22d8fe366\" (UID: \"6e1d475b-2f4c-4224-8ec9-ebc22d8fe366\") " Jul 19 08:32:19.614701 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:19.614711 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e1d475b-2f4c-4224-8ec9-ebc22d8fe366-istio-data\") pod \"6e1d475b-2f4c-4224-8ec9-ebc22d8fe366\" (UID: \"6e1d475b-2f4c-4224-8ec9-ebc22d8fe366\") " Jul 19 08:32:19.614982 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:19.614769 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e1d475b-2f4c-4224-8ec9-ebc22d8fe366-istiod-ca-cert\") pod \"6e1d475b-2f4c-4224-8ec9-ebc22d8fe366\" (UID: \"6e1d475b-2f4c-4224-8ec9-ebc22d8fe366\") " Jul 19 08:32:19.614982 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:19.614805 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e1d475b-2f4c-4224-8ec9-ebc22d8fe366-credential-socket\") pod \"6e1d475b-2f4c-4224-8ec9-ebc22d8fe366\" (UID: \"6e1d475b-2f4c-4224-8ec9-ebc22d8fe366\") " Jul 19 08:32:19.614982 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:19.614828 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e1d475b-2f4c-4224-8ec9-ebc22d8fe366-istio-podinfo\") pod \"6e1d475b-2f4c-4224-8ec9-ebc22d8fe366\" (UID: \"6e1d475b-2f4c-4224-8ec9-ebc22d8fe366\") " Jul 19 08:32:19.614982 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:19.614954 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jvrbl\" (UniqueName: \"kubernetes.io/projected/6e1d475b-2f4c-4224-8ec9-ebc22d8fe366-kube-api-access-jvrbl\") pod \"6e1d475b-2f4c-4224-8ec9-ebc22d8fe366\" (UID: \"6e1d475b-2f4c-4224-8ec9-ebc22d8fe366\") " Jul 19 08:32:19.615169 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:19.615025 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e1d475b-2f4c-4224-8ec9-ebc22d8fe366-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6e1d475b-2f4c-4224-8ec9-ebc22d8fe366" (UID: "6e1d475b-2f4c-4224-8ec9-ebc22d8fe366"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 08:32:19.615169 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:19.615067 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e1d475b-2f4c-4224-8ec9-ebc22d8fe366-istio-envoy\") pod \"6e1d475b-2f4c-4224-8ec9-ebc22d8fe366\" (UID: \"6e1d475b-2f4c-4224-8ec9-ebc22d8fe366\") " Jul 19 08:32:19.615169 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:19.615116 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e1d475b-2f4c-4224-8ec9-ebc22d8fe366-workload-certs\") pod \"6e1d475b-2f4c-4224-8ec9-ebc22d8fe366\" (UID: \"6e1d475b-2f4c-4224-8ec9-ebc22d8fe366\") " Jul 19 08:32:19.615169 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:19.615136 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e1d475b-2f4c-4224-8ec9-ebc22d8fe366-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6e1d475b-2f4c-4224-8ec9-ebc22d8fe366" (UID: "6e1d475b-2f4c-4224-8ec9-ebc22d8fe366"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 08:32:19.615169 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:19.615145 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e1d475b-2f4c-4224-8ec9-ebc22d8fe366-istio-token\") pod \"6e1d475b-2f4c-4224-8ec9-ebc22d8fe366\" (UID: \"6e1d475b-2f4c-4224-8ec9-ebc22d8fe366\") " Jul 19 08:32:19.615421 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:19.615373 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e1d475b-2f4c-4224-8ec9-ebc22d8fe366-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6e1d475b-2f4c-4224-8ec9-ebc22d8fe366" (UID: "6e1d475b-2f4c-4224-8ec9-ebc22d8fe366"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 08:32:19.615421 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:19.615397 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e1d475b-2f4c-4224-8ec9-ebc22d8fe366-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6e1d475b-2f4c-4224-8ec9-ebc22d8fe366" (UID: "6e1d475b-2f4c-4224-8ec9-ebc22d8fe366"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 08:32:19.615571 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:19.615448 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e1d475b-2f4c-4224-8ec9-ebc22d8fe366-workload-socket\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:32:19.615571 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:19.615465 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e1d475b-2f4c-4224-8ec9-ebc22d8fe366-istiod-ca-cert\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:32:19.615571 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:19.615479 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e1d475b-2f4c-4224-8ec9-ebc22d8fe366-credential-socket\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:32:19.615713 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:19.615655 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e1d475b-2f4c-4224-8ec9-ebc22d8fe366-istio-data" (OuterVolumeSpecName: "istio-data") pod "6e1d475b-2f4c-4224-8ec9-ebc22d8fe366" (UID: "6e1d475b-2f4c-4224-8ec9-ebc22d8fe366"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 08:32:19.617228 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:19.617207 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e1d475b-2f4c-4224-8ec9-ebc22d8fe366-kube-api-access-jvrbl" (OuterVolumeSpecName: "kube-api-access-jvrbl") pod "6e1d475b-2f4c-4224-8ec9-ebc22d8fe366" (UID: "6e1d475b-2f4c-4224-8ec9-ebc22d8fe366"). InnerVolumeSpecName "kube-api-access-jvrbl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 08:32:19.617351 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:19.617276 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e1d475b-2f4c-4224-8ec9-ebc22d8fe366-istio-token" (OuterVolumeSpecName: "istio-token") pod "6e1d475b-2f4c-4224-8ec9-ebc22d8fe366" (UID: "6e1d475b-2f4c-4224-8ec9-ebc22d8fe366"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 08:32:19.617593 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:19.617574 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e1d475b-2f4c-4224-8ec9-ebc22d8fe366-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6e1d475b-2f4c-4224-8ec9-ebc22d8fe366" (UID: "6e1d475b-2f4c-4224-8ec9-ebc22d8fe366"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 08:32:19.617694 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:19.617677 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6e1d475b-2f4c-4224-8ec9-ebc22d8fe366-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6e1d475b-2f4c-4224-8ec9-ebc22d8fe366" (UID: "6e1d475b-2f4c-4224-8ec9-ebc22d8fe366"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 19 08:32:19.716050 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:19.716016 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvrbl\" (UniqueName: \"kubernetes.io/projected/6e1d475b-2f4c-4224-8ec9-ebc22d8fe366-kube-api-access-jvrbl\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:32:19.716050 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:19.716047 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e1d475b-2f4c-4224-8ec9-ebc22d8fe366-istio-envoy\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:32:19.716050 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:19.716057 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e1d475b-2f4c-4224-8ec9-ebc22d8fe366-workload-certs\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:32:19.716316 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:19.716065 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e1d475b-2f4c-4224-8ec9-ebc22d8fe366-istio-token\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:32:19.716316 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:19.716073 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e1d475b-2f4c-4224-8ec9-ebc22d8fe366-istio-data\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:32:19.716316 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:19.716080 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e1d475b-2f4c-4224-8ec9-ebc22d8fe366-istio-podinfo\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:32:20.086349 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:20.086231 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5da88c-667c58dfd6-74j86" event={"ID":"9619b8aa-97a1-4901-96db-302e4502d2e6","Type":"ContainerStarted","Data":"8e8135d88a71183df81813d7e962b4f37492e344bd612cd85cecb99fcbca09d5"} Jul 19 08:32:20.086828 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:20.086402 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5da88c-667c58dfd6-74j86" Jul 19 08:32:20.087987 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:20.087954 2574 generic.go:356] "Generic (PLEG): container finished" podID="6e1d475b-2f4c-4224-8ec9-ebc22d8fe366" containerID="600d91863bb7ff37d8464d5d856b0e6aeb8f420a04d10f1b68dc4df73ff717a6" exitCode=0 Jul 19 08:32:20.088105 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:20.088014 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-916a44-openshift-default-56bfbfdfbf-v5mkx" Jul 19 08:32:20.088105 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:20.088042 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-916a44-openshift-default-56bfbfdfbf-v5mkx" event={"ID":"6e1d475b-2f4c-4224-8ec9-ebc22d8fe366","Type":"ContainerDied","Data":"600d91863bb7ff37d8464d5d856b0e6aeb8f420a04d10f1b68dc4df73ff717a6"} Jul 19 08:32:20.088105 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:20.088085 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-916a44-openshift-default-56bfbfdfbf-v5mkx" event={"ID":"6e1d475b-2f4c-4224-8ec9-ebc22d8fe366","Type":"ContainerDied","Data":"0a8e155b3cd030ff95866ae8eb07b4c7dfd1786f0f67abcf68a9f45785a0acd0"} Jul 19 08:32:20.088216 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:20.088111 2574 scope.go:117] "RemoveContainer" containerID="600d91863bb7ff37d8464d5d856b0e6aeb8f420a04d10f1b68dc4df73ff717a6" Jul 19 08:32:20.097972 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:20.097957 2574 scope.go:117] "RemoveContainer" containerID="70a4a174cb4c9d7dfaa923139d33103ea9b33f0398584e1dda8186a3019cfa48" Jul 19 08:32:20.115165 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:20.115142 2574 scope.go:117] "RemoveContainer" containerID="600d91863bb7ff37d8464d5d856b0e6aeb8f420a04d10f1b68dc4df73ff717a6" Jul 19 08:32:20.115407 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:32:20.115389 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"600d91863bb7ff37d8464d5d856b0e6aeb8f420a04d10f1b68dc4df73ff717a6\": container with ID starting with 600d91863bb7ff37d8464d5d856b0e6aeb8f420a04d10f1b68dc4df73ff717a6 not found: ID does not exist" containerID="600d91863bb7ff37d8464d5d856b0e6aeb8f420a04d10f1b68dc4df73ff717a6" Jul 19 08:32:20.115444 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:20.115417 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"600d91863bb7ff37d8464d5d856b0e6aeb8f420a04d10f1b68dc4df73ff717a6"} err="failed to get container status \"600d91863bb7ff37d8464d5d856b0e6aeb8f420a04d10f1b68dc4df73ff717a6\": rpc error: code = NotFound desc = could not find container \"600d91863bb7ff37d8464d5d856b0e6aeb8f420a04d10f1b68dc4df73ff717a6\": container with ID starting with 600d91863bb7ff37d8464d5d856b0e6aeb8f420a04d10f1b68dc4df73ff717a6 not found: ID does not exist" Jul 19 08:32:20.115444 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:20.115437 2574 scope.go:117] "RemoveContainer" containerID="70a4a174cb4c9d7dfaa923139d33103ea9b33f0398584e1dda8186a3019cfa48" Jul 19 08:32:20.115694 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:32:20.115677 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70a4a174cb4c9d7dfaa923139d33103ea9b33f0398584e1dda8186a3019cfa48\": container with ID starting with 70a4a174cb4c9d7dfaa923139d33103ea9b33f0398584e1dda8186a3019cfa48 not found: ID does not exist" containerID="70a4a174cb4c9d7dfaa923139d33103ea9b33f0398584e1dda8186a3019cfa48" Jul 19 08:32:20.115729 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:20.115701 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70a4a174cb4c9d7dfaa923139d33103ea9b33f0398584e1dda8186a3019cfa48"} err="failed to get container status \"70a4a174cb4c9d7dfaa923139d33103ea9b33f0398584e1dda8186a3019cfa48\": rpc error: code = NotFound desc = could not find container \"70a4a174cb4c9d7dfaa923139d33103ea9b33f0398584e1dda8186a3019cfa48\": container with ID starting with 70a4a174cb4c9d7dfaa923139d33103ea9b33f0398584e1dda8186a3019cfa48 not found: ID does not exist" Jul 19 08:32:20.118801 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:20.118757 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5da88c-667c58dfd6-74j86" podStartSLOduration=1.583417153 podStartE2EDuration="2.118741896s" totalImagesPullingTime="535.324743ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 08:32:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:32:20.106318341 +0000 UTC m=+3006.460627941" watchObservedRunningTime="2026-07-19 08:32:20.118741896 +0000 UTC m=+3006.473051498" Jul 19 08:32:20.119407 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:20.119390 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-916a44-openshift-default-56bfbfdfbf-v5mkx"] Jul 19 08:32:20.123312 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:20.123289 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-916a44-openshift-default-56bfbfdfbf-v5mkx"] Jul 19 08:32:20.559386 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:20.559350 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e1d475b-2f4c-4224-8ec9-ebc22d8fe366" path="/var/lib/kubelet/pods/6e1d475b-2f4c-4224-8ec9-ebc22d8fe366/volumes" Jul 19 08:32:22.467386 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:22.467351 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:32:22.928734 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:22.928633 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:32:23.169054 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:23.169011 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-2d2deb-openshift-default-79d58c4cbc-km8cv"] Jul 19 08:32:23.169435 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:23.169414 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e1d475b-2f4c-4224-8ec9-ebc22d8fe366" containerName="istio-proxy" Jul 19 08:32:23.169435 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:23.169432 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e1d475b-2f4c-4224-8ec9-ebc22d8fe366" containerName="istio-proxy" Jul 19 08:32:23.169664 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:23.169455 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e1d475b-2f4c-4224-8ec9-ebc22d8fe366" containerName="istio-proxy" Jul 19 08:32:23.169664 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:23.169464 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e1d475b-2f4c-4224-8ec9-ebc22d8fe366" containerName="istio-proxy" Jul 19 08:32:23.169664 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:23.169604 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e1d475b-2f4c-4224-8ec9-ebc22d8fe366" containerName="istio-proxy" Jul 19 08:32:23.169664 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:23.169618 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e1d475b-2f4c-4224-8ec9-ebc22d8fe366" containerName="istio-proxy" Jul 19 08:32:23.174556 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:23.174514 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2d2deb-openshift-default-79d58c4cbc-km8cv" Jul 19 08:32:23.179020 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:23.178955 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-2d2deb-openshift-default-dockercfg-fr45l\"" Jul 19 08:32:23.186259 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:23.186232 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-2d2deb-openshift-default-79d58c4cbc-km8cv"] Jul 19 08:32:23.198804 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:23.198773 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:32:23.273099 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:23.273059 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 08:32:23.350300 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:23.350262 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k52jn\" (UniqueName: \"kubernetes.io/projected/02d3ea7e-ea37-4970-8afa-2040144cbb89-kube-api-access-k52jn\") pod \"e2e-api-b-2d2deb-openshift-default-79d58c4cbc-km8cv\" (UID: \"02d3ea7e-ea37-4970-8afa-2040144cbb89\") " pod="openshift-ingress/e2e-api-b-2d2deb-openshift-default-79d58c4cbc-km8cv" Jul 19 08:32:23.350300 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:23.350298 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02d3ea7e-ea37-4970-8afa-2040144cbb89-istio-token\") pod \"e2e-api-b-2d2deb-openshift-default-79d58c4cbc-km8cv\" (UID: \"02d3ea7e-ea37-4970-8afa-2040144cbb89\") " pod="openshift-ingress/e2e-api-b-2d2deb-openshift-default-79d58c4cbc-km8cv" Jul 19 08:32:23.350565 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:23.350323 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02d3ea7e-ea37-4970-8afa-2040144cbb89-istio-podinfo\") pod \"e2e-api-b-2d2deb-openshift-default-79d58c4cbc-km8cv\" (UID: \"02d3ea7e-ea37-4970-8afa-2040144cbb89\") " pod="openshift-ingress/e2e-api-b-2d2deb-openshift-default-79d58c4cbc-km8cv" Jul 19 08:32:23.350565 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:23.350380 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02d3ea7e-ea37-4970-8afa-2040144cbb89-istiod-ca-cert\") pod \"e2e-api-b-2d2deb-openshift-default-79d58c4cbc-km8cv\" (UID: \"02d3ea7e-ea37-4970-8afa-2040144cbb89\") " pod="openshift-ingress/e2e-api-b-2d2deb-openshift-default-79d58c4cbc-km8cv" Jul 19 08:32:23.350565 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:23.350451 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02d3ea7e-ea37-4970-8afa-2040144cbb89-credential-socket\") pod \"e2e-api-b-2d2deb-openshift-default-79d58c4cbc-km8cv\" (UID: \"02d3ea7e-ea37-4970-8afa-2040144cbb89\") " pod="openshift-ingress/e2e-api-b-2d2deb-openshift-default-79d58c4cbc-km8cv" Jul 19 08:32:23.350565 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:23.350484 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02d3ea7e-ea37-4970-8afa-2040144cbb89-workload-certs\") pod \"e2e-api-b-2d2deb-openshift-default-79d58c4cbc-km8cv\" (UID: \"02d3ea7e-ea37-4970-8afa-2040144cbb89\") " pod="openshift-ingress/e2e-api-b-2d2deb-openshift-default-79d58c4cbc-km8cv" Jul 19 08:32:23.350734 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:23.350569 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02d3ea7e-ea37-4970-8afa-2040144cbb89-istio-envoy\") pod \"e2e-api-b-2d2deb-openshift-default-79d58c4cbc-km8cv\" (UID: \"02d3ea7e-ea37-4970-8afa-2040144cbb89\") " pod="openshift-ingress/e2e-api-b-2d2deb-openshift-default-79d58c4cbc-km8cv" Jul 19 08:32:23.350734 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:23.350609 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02d3ea7e-ea37-4970-8afa-2040144cbb89-workload-socket\") pod \"e2e-api-b-2d2deb-openshift-default-79d58c4cbc-km8cv\" (UID: \"02d3ea7e-ea37-4970-8afa-2040144cbb89\") " pod="openshift-ingress/e2e-api-b-2d2deb-openshift-default-79d58c4cbc-km8cv" Jul 19 08:32:23.350734 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:23.350631 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02d3ea7e-ea37-4970-8afa-2040144cbb89-istio-data\") pod \"e2e-api-b-2d2deb-openshift-default-79d58c4cbc-km8cv\" (UID: \"02d3ea7e-ea37-4970-8afa-2040144cbb89\") " pod="openshift-ingress/e2e-api-b-2d2deb-openshift-default-79d58c4cbc-km8cv" Jul 19 08:32:23.451471 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:23.451373 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k52jn\" (UniqueName: \"kubernetes.io/projected/02d3ea7e-ea37-4970-8afa-2040144cbb89-kube-api-access-k52jn\") pod \"e2e-api-b-2d2deb-openshift-default-79d58c4cbc-km8cv\" (UID: \"02d3ea7e-ea37-4970-8afa-2040144cbb89\") " pod="openshift-ingress/e2e-api-b-2d2deb-openshift-default-79d58c4cbc-km8cv" Jul 19 08:32:23.451471 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:23.451424 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02d3ea7e-ea37-4970-8afa-2040144cbb89-istio-token\") pod \"e2e-api-b-2d2deb-openshift-default-79d58c4cbc-km8cv\" (UID: \"02d3ea7e-ea37-4970-8afa-2040144cbb89\") " pod="openshift-ingress/e2e-api-b-2d2deb-openshift-default-79d58c4cbc-km8cv" Jul 19 08:32:23.451471 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:23.451462 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02d3ea7e-ea37-4970-8afa-2040144cbb89-istio-podinfo\") pod \"e2e-api-b-2d2deb-openshift-default-79d58c4cbc-km8cv\" (UID: \"02d3ea7e-ea37-4970-8afa-2040144cbb89\") " pod="openshift-ingress/e2e-api-b-2d2deb-openshift-default-79d58c4cbc-km8cv" Jul 19 08:32:23.451775 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:23.451491 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02d3ea7e-ea37-4970-8afa-2040144cbb89-istiod-ca-cert\") pod \"e2e-api-b-2d2deb-openshift-default-79d58c4cbc-km8cv\" (UID: \"02d3ea7e-ea37-4970-8afa-2040144cbb89\") " pod="openshift-ingress/e2e-api-b-2d2deb-openshift-default-79d58c4cbc-km8cv" Jul 19 08:32:23.451876 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:23.451851 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02d3ea7e-ea37-4970-8afa-2040144cbb89-credential-socket\") pod \"e2e-api-b-2d2deb-openshift-default-79d58c4cbc-km8cv\" (UID: \"02d3ea7e-ea37-4970-8afa-2040144cbb89\") " pod="openshift-ingress/e2e-api-b-2d2deb-openshift-default-79d58c4cbc-km8cv" Jul 19 08:32:23.451944 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:23.451918 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02d3ea7e-ea37-4970-8afa-2040144cbb89-workload-certs\") pod \"e2e-api-b-2d2deb-openshift-default-79d58c4cbc-km8cv\" (UID: \"02d3ea7e-ea37-4970-8afa-2040144cbb89\") " pod="openshift-ingress/e2e-api-b-2d2deb-openshift-default-79d58c4cbc-km8cv" Jul 19 08:32:23.452070 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:23.451994 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02d3ea7e-ea37-4970-8afa-2040144cbb89-istio-envoy\") pod \"e2e-api-b-2d2deb-openshift-default-79d58c4cbc-km8cv\" (UID: \"02d3ea7e-ea37-4970-8afa-2040144cbb89\") " pod="openshift-ingress/e2e-api-b-2d2deb-openshift-default-79d58c4cbc-km8cv" Jul 19 08:32:23.452070 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:23.452044 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02d3ea7e-ea37-4970-8afa-2040144cbb89-workload-socket\") pod \"e2e-api-b-2d2deb-openshift-default-79d58c4cbc-km8cv\" (UID: \"02d3ea7e-ea37-4970-8afa-2040144cbb89\") " pod="openshift-ingress/e2e-api-b-2d2deb-openshift-default-79d58c4cbc-km8cv" Jul 19 08:32:23.452182 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:23.452072 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02d3ea7e-ea37-4970-8afa-2040144cbb89-istio-data\") pod \"e2e-api-b-2d2deb-openshift-default-79d58c4cbc-km8cv\" (UID: \"02d3ea7e-ea37-4970-8afa-2040144cbb89\") " pod="openshift-ingress/e2e-api-b-2d2deb-openshift-default-79d58c4cbc-km8cv" Jul 19 08:32:23.452829 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:23.452301 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02d3ea7e-ea37-4970-8afa-2040144cbb89-istiod-ca-cert\") pod \"e2e-api-b-2d2deb-openshift-default-79d58c4cbc-km8cv\" (UID: \"02d3ea7e-ea37-4970-8afa-2040144cbb89\") " pod="openshift-ingress/e2e-api-b-2d2deb-openshift-default-79d58c4cbc-km8cv" Jul 19 08:32:23.452829 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:23.452317 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02d3ea7e-ea37-4970-8afa-2040144cbb89-credential-socket\") pod \"e2e-api-b-2d2deb-openshift-default-79d58c4cbc-km8cv\" (UID: \"02d3ea7e-ea37-4970-8afa-2040144cbb89\") " pod="openshift-ingress/e2e-api-b-2d2deb-openshift-default-79d58c4cbc-km8cv" Jul 19 08:32:23.452829 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:23.452492 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02d3ea7e-ea37-4970-8afa-2040144cbb89-istio-data\") pod \"e2e-api-b-2d2deb-openshift-default-79d58c4cbc-km8cv\" (UID: \"02d3ea7e-ea37-4970-8afa-2040144cbb89\") " pod="openshift-ingress/e2e-api-b-2d2deb-openshift-default-79d58c4cbc-km8cv" Jul 19 08:32:23.452829 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:23.452610 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02d3ea7e-ea37-4970-8afa-2040144cbb89-workload-certs\") pod \"e2e-api-b-2d2deb-openshift-default-79d58c4cbc-km8cv\" (UID: \"02d3ea7e-ea37-4970-8afa-2040144cbb89\") " pod="openshift-ingress/e2e-api-b-2d2deb-openshift-default-79d58c4cbc-km8cv" Jul 19 08:32:23.453146 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:23.452860 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02d3ea7e-ea37-4970-8afa-2040144cbb89-workload-socket\") pod \"e2e-api-b-2d2deb-openshift-default-79d58c4cbc-km8cv\" (UID: \"02d3ea7e-ea37-4970-8afa-2040144cbb89\") " pod="openshift-ingress/e2e-api-b-2d2deb-openshift-default-79d58c4cbc-km8cv" Jul 19 08:32:23.455068 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:23.455042 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02d3ea7e-ea37-4970-8afa-2040144cbb89-istio-envoy\") pod \"e2e-api-b-2d2deb-openshift-default-79d58c4cbc-km8cv\" (UID: \"02d3ea7e-ea37-4970-8afa-2040144cbb89\") " pod="openshift-ingress/e2e-api-b-2d2deb-openshift-default-79d58c4cbc-km8cv" Jul 19 08:32:23.455388 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:23.455364 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02d3ea7e-ea37-4970-8afa-2040144cbb89-istio-podinfo\") pod \"e2e-api-b-2d2deb-openshift-default-79d58c4cbc-km8cv\" (UID: \"02d3ea7e-ea37-4970-8afa-2040144cbb89\") " pod="openshift-ingress/e2e-api-b-2d2deb-openshift-default-79d58c4cbc-km8cv" Jul 19 08:32:23.459589 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:23.459560 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02d3ea7e-ea37-4970-8afa-2040144cbb89-istio-token\") pod \"e2e-api-b-2d2deb-openshift-default-79d58c4cbc-km8cv\" (UID: \"02d3ea7e-ea37-4970-8afa-2040144cbb89\") " pod="openshift-ingress/e2e-api-b-2d2deb-openshift-default-79d58c4cbc-km8cv" Jul 19 08:32:23.459784 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:23.459758 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k52jn\" (UniqueName: \"kubernetes.io/projected/02d3ea7e-ea37-4970-8afa-2040144cbb89-kube-api-access-k52jn\") pod \"e2e-api-b-2d2deb-openshift-default-79d58c4cbc-km8cv\" (UID: \"02d3ea7e-ea37-4970-8afa-2040144cbb89\") " pod="openshift-ingress/e2e-api-b-2d2deb-openshift-default-79d58c4cbc-km8cv" Jul 19 08:32:23.486497 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:23.486467 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2d2deb-openshift-default-79d58c4cbc-km8cv" Jul 19 08:32:23.659931 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:23.659893 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:32:23.857596 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:23.857323 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-2d2deb-openshift-default-79d58c4cbc-km8cv"] Jul 19 08:32:23.860735 ip-10-0-136-70 kubenswrapper[2574]: W0719 08:32:23.860707 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02d3ea7e_ea37_4970_8afa_2040144cbb89.slice/crio-4cc7e42721f9d2c012e8a07854e2c2393a8fba1dcde29fb02b0f24eb27f88c8a WatchSource:0}: Error finding container 4cc7e42721f9d2c012e8a07854e2c2393a8fba1dcde29fb02b0f24eb27f88c8a: Status 404 returned error can't find the container with id 4cc7e42721f9d2c012e8a07854e2c2393a8fba1dcde29fb02b0f24eb27f88c8a Jul 19 08:32:23.863409 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:23.863376 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 19 08:32:23.863506 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:23.863459 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 19 08:32:23.863621 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:23.863504 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 19 08:32:24.114042 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:24.113934 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2d2deb-openshift-default-79d58c4cbc-km8cv" event={"ID":"02d3ea7e-ea37-4970-8afa-2040144cbb89","Type":"ContainerStarted","Data":"175049f9d0eef6215d328c757144a6b0867f9f0e349b94982112030ed167a0a9"} Jul 19 08:32:24.114333 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:24.114315 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2d2deb-openshift-default-79d58c4cbc-km8cv" event={"ID":"02d3ea7e-ea37-4970-8afa-2040144cbb89","Type":"ContainerStarted","Data":"4cc7e42721f9d2c012e8a07854e2c2393a8fba1dcde29fb02b0f24eb27f88c8a"} Jul 19 08:32:24.139870 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:24.139743 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-2d2deb-openshift-default-79d58c4cbc-km8cv" podStartSLOduration=1.139717223 podStartE2EDuration="1.139717223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 08:32:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:32:24.138333016 +0000 UTC m=+3010.492642617" watchObservedRunningTime="2026-07-19 08:32:24.139717223 +0000 UTC m=+3010.494026826" Jul 19 08:32:24.486784 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:24.486753 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-2d2deb-openshift-default-79d58c4cbc-km8cv" Jul 19 08:32:24.492448 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:24.492416 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-2d2deb-openshift-default-79d58c4cbc-km8cv" Jul 19 08:32:25.121792 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:25.121753 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-2d2deb-openshift-default-79d58c4cbc-km8cv" Jul 19 08:32:25.130094 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:25.130058 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-2d2deb-openshift-default-79d58c4cbc-km8cv" Jul 19 08:32:26.047312 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:26.047266 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:32:26.100225 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:26.100197 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5da88c-667c58dfd6-74j86" Jul 19 08:32:26.189375 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:26.189304 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:32:26.210057 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:26.210026 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2d2deb-6b8bf7f755-s99h9"] Jul 19 08:32:26.213869 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:26.213844 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2d2deb-6b8bf7f755-s99h9" Jul 19 08:32:26.216985 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:26.216951 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-2d2deb-serving-cert\"" Jul 19 08:32:26.264487 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:26.264438 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:32:26.265048 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:26.265026 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2d2deb-6b8bf7f755-s99h9"] Jul 19 08:32:26.278226 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:26.278201 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/001b4a01-654f-422f-bd4f-0e8350fdbedb-maas-api-tls\") pod \"maas-api-e2e-api-b-2d2deb-6b8bf7f755-s99h9\" (UID: \"001b4a01-654f-422f-bd4f-0e8350fdbedb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2d2deb-6b8bf7f755-s99h9" Jul 19 08:32:26.278372 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:26.278258 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/001b4a01-654f-422f-bd4f-0e8350fdbedb-openshift-service-ca\") pod \"maas-api-e2e-api-b-2d2deb-6b8bf7f755-s99h9\" (UID: \"001b4a01-654f-422f-bd4f-0e8350fdbedb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2d2deb-6b8bf7f755-s99h9" Jul 19 08:32:26.278372 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:26.278354 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2x59\" (UniqueName: \"kubernetes.io/projected/001b4a01-654f-422f-bd4f-0e8350fdbedb-kube-api-access-h2x59\") pod \"maas-api-e2e-api-b-2d2deb-6b8bf7f755-s99h9\" (UID: \"001b4a01-654f-422f-bd4f-0e8350fdbedb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2d2deb-6b8bf7f755-s99h9" Jul 19 08:32:26.379209 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:26.379104 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/001b4a01-654f-422f-bd4f-0e8350fdbedb-maas-api-tls\") pod \"maas-api-e2e-api-b-2d2deb-6b8bf7f755-s99h9\" (UID: \"001b4a01-654f-422f-bd4f-0e8350fdbedb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2d2deb-6b8bf7f755-s99h9" Jul 19 08:32:26.379209 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:26.379151 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/001b4a01-654f-422f-bd4f-0e8350fdbedb-openshift-service-ca\") pod \"maas-api-e2e-api-b-2d2deb-6b8bf7f755-s99h9\" (UID: \"001b4a01-654f-422f-bd4f-0e8350fdbedb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2d2deb-6b8bf7f755-s99h9" Jul 19 08:32:26.379209 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:26.379204 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2x59\" (UniqueName: \"kubernetes.io/projected/001b4a01-654f-422f-bd4f-0e8350fdbedb-kube-api-access-h2x59\") pod \"maas-api-e2e-api-b-2d2deb-6b8bf7f755-s99h9\" (UID: \"001b4a01-654f-422f-bd4f-0e8350fdbedb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2d2deb-6b8bf7f755-s99h9" Jul 19 08:32:26.379511 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:32:26.379254 2574 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-2d2deb-serving-cert: secret "maas-api-e2e-api-b-2d2deb-serving-cert" not found Jul 19 08:32:26.379511 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:32:26.379340 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/001b4a01-654f-422f-bd4f-0e8350fdbedb-maas-api-tls podName:001b4a01-654f-422f-bd4f-0e8350fdbedb nodeName:}" failed. No retries permitted until 2026-07-19 08:32:26.87931719 +0000 UTC m=+3013.233626770 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/001b4a01-654f-422f-bd4f-0e8350fdbedb-maas-api-tls") pod "maas-api-e2e-api-b-2d2deb-6b8bf7f755-s99h9" (UID: "001b4a01-654f-422f-bd4f-0e8350fdbedb") : secret "maas-api-e2e-api-b-2d2deb-serving-cert" not found Jul 19 08:32:26.379932 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:26.379912 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/001b4a01-654f-422f-bd4f-0e8350fdbedb-openshift-service-ca\") pod \"maas-api-e2e-api-b-2d2deb-6b8bf7f755-s99h9\" (UID: \"001b4a01-654f-422f-bd4f-0e8350fdbedb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2d2deb-6b8bf7f755-s99h9" Jul 19 08:32:26.388315 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:26.388280 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2x59\" (UniqueName: \"kubernetes.io/projected/001b4a01-654f-422f-bd4f-0e8350fdbedb-kube-api-access-h2x59\") pod \"maas-api-e2e-api-b-2d2deb-6b8bf7f755-s99h9\" (UID: \"001b4a01-654f-422f-bd4f-0e8350fdbedb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2d2deb-6b8bf7f755-s99h9" Jul 19 08:32:26.602063 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:26.602024 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:32:26.814688 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:26.814650 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:32:26.882097 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:26.882064 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/001b4a01-654f-422f-bd4f-0e8350fdbedb-maas-api-tls\") pod \"maas-api-e2e-api-b-2d2deb-6b8bf7f755-s99h9\" (UID: \"001b4a01-654f-422f-bd4f-0e8350fdbedb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2d2deb-6b8bf7f755-s99h9" Jul 19 08:32:26.884569 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:26.884535 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/001b4a01-654f-422f-bd4f-0e8350fdbedb-maas-api-tls\") pod \"maas-api-e2e-api-b-2d2deb-6b8bf7f755-s99h9\" (UID: \"001b4a01-654f-422f-bd4f-0e8350fdbedb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2d2deb-6b8bf7f755-s99h9" Jul 19 08:32:26.915408 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:26.915364 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:32:27.161119 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:27.161025 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2d2deb-6b8bf7f755-s99h9" Jul 19 08:32:27.346571 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:27.342273 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2d2deb-6b8bf7f755-s99h9"] Jul 19 08:32:27.351744 ip-10-0-136-70 kubenswrapper[2574]: W0719 08:32:27.351711 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod001b4a01_654f_422f_bd4f_0e8350fdbedb.slice/crio-2b31ad29b71a472889637ea9a02cc4f725d493233e3ef27aabe45d677b8290a7 WatchSource:0}: Error finding container 2b31ad29b71a472889637ea9a02cc4f725d493233e3ef27aabe45d677b8290a7: Status 404 returned error can't find the container with id 2b31ad29b71a472889637ea9a02cc4f725d493233e3ef27aabe45d677b8290a7 Jul 19 08:32:28.153017 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:28.152853 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2d2deb-6b8bf7f755-s99h9" event={"ID":"001b4a01-654f-422f-bd4f-0e8350fdbedb","Type":"ContainerStarted","Data":"1ce000fbbc7e18fb7ad0e97b081723d61dfde9688343af96975737ac155598bf"} Jul 19 08:32:28.153017 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:28.152923 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2d2deb-6b8bf7f755-s99h9" event={"ID":"001b4a01-654f-422f-bd4f-0e8350fdbedb","Type":"ContainerStarted","Data":"2b31ad29b71a472889637ea9a02cc4f725d493233e3ef27aabe45d677b8290a7"} Jul 19 08:32:28.153513 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:28.153400 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2d2deb-6b8bf7f755-s99h9" Jul 19 08:32:28.169901 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:28.169852 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2d2deb-6b8bf7f755-s99h9" podStartSLOduration=1.862793451 podStartE2EDuration="2.169832625s" totalImagesPullingTime="307.039174ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 08:32:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:32:28.169566617 +0000 UTC m=+3014.523876219" watchObservedRunningTime="2026-07-19 08:32:28.169832625 +0000 UTC m=+3014.524142225" Jul 19 08:32:30.511064 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:30.511019 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:32:32.067464 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:32.067424 2574 patch_prober.go:28] interesting pod/e2e-api-a-5da88c-openshift-default-5c89fb85d5-s9flw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Jul 19 08:32:32.067953 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:32.067494 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-5da88c-openshift-default-5c89fb85d5-s9flw" podUID="310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Jul 19 08:32:32.175427 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:32.175392 2574 generic.go:356] "Generic (PLEG): container finished" podID="310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f" containerID="cc1ba68e15fc8c56afbf0bcdde4e6699e5de322d39e74f49795ea38c9b4484f2" exitCode=0 Jul 19 08:32:32.175616 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:32.175438 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5da88c-openshift-default-5c89fb85d5-s9flw" event={"ID":"310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f","Type":"ContainerDied","Data":"cc1ba68e15fc8c56afbf0bcdde4e6699e5de322d39e74f49795ea38c9b4484f2"} Jul 19 08:32:32.175870 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:32.175856 2574 scope.go:117] "RemoveContainer" containerID="cc1ba68e15fc8c56afbf0bcdde4e6699e5de322d39e74f49795ea38c9b4484f2" Jul 19 08:32:32.176847 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:32.176820 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 19 08:32:32.176907 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:32.176890 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 19 08:32:32.176952 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:32.176922 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 19 08:32:33.183145 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:33.183107 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5da88c-openshift-default-5c89fb85d5-s9flw" event={"ID":"310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f","Type":"ContainerStarted","Data":"894a94f1b118ca19e07d363f0de7bb5a787564c123cc458748dfb68b611f8cba"} Jul 19 08:32:33.393108 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:33.393069 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-5da88c-openshift-default-5c89fb85d5-s9flw" Jul 19 08:32:33.393295 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:33.393177 2574 patch_prober.go:28] interesting pod/e2e-api-a-5da88c-openshift-default-5c89fb85d5-s9flw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Jul 19 08:32:33.393295 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:33.393212 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5da88c-openshift-default-5c89fb85d5-s9flw" podUID="310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Jul 19 08:32:34.171310 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:34.171273 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2d2deb-6b8bf7f755-s99h9" Jul 19 08:32:34.392925 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:34.392883 2574 patch_prober.go:28] interesting pod/e2e-api-a-5da88c-openshift-default-5c89fb85d5-s9flw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Jul 19 08:32:34.393329 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:34.392956 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5da88c-openshift-default-5c89fb85d5-s9flw" podUID="310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Jul 19 08:32:35.393185 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:35.393148 2574 patch_prober.go:28] interesting pod/e2e-api-a-5da88c-openshift-default-5c89fb85d5-s9flw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Jul 19 08:32:35.393618 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:35.393199 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5da88c-openshift-default-5c89fb85d5-s9flw" podUID="310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Jul 19 08:32:36.392451 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:36.392415 2574 patch_prober.go:28] interesting pod/e2e-api-a-5da88c-openshift-default-5c89fb85d5-s9flw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Jul 19 08:32:36.392681 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:36.392470 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5da88c-openshift-default-5c89fb85d5-s9flw" podUID="310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Jul 19 08:32:37.396910 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:37.396869 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-5da88c-openshift-default-5c89fb85d5-s9flw" Jul 19 08:32:37.397396 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:37.397331 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-5da88c-openshift-default-5c89fb85d5-s9flw" Jul 19 08:32:37.398129 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:37.398109 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-5da88c-openshift-default-5c89fb85d5-s9flw" Jul 19 08:32:39.215756 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:39.215722 2574 generic.go:356] "Generic (PLEG): container finished" podID="02d3ea7e-ea37-4970-8afa-2040144cbb89" containerID="175049f9d0eef6215d328c757144a6b0867f9f0e349b94982112030ed167a0a9" exitCode=0 Jul 19 08:32:39.216211 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:39.215797 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2d2deb-openshift-default-79d58c4cbc-km8cv" event={"ID":"02d3ea7e-ea37-4970-8afa-2040144cbb89","Type":"ContainerDied","Data":"175049f9d0eef6215d328c757144a6b0867f9f0e349b94982112030ed167a0a9"} Jul 19 08:32:39.216304 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:39.216287 2574 scope.go:117] "RemoveContainer" containerID="175049f9d0eef6215d328c757144a6b0867f9f0e349b94982112030ed167a0a9" Jul 19 08:32:39.217284 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:39.217250 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 19 08:32:39.217355 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:39.217337 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 19 08:32:39.217401 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:39.217388 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 19 08:32:39.275837 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:39.275804 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:32:39.375887 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:39.375848 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:32:40.122399 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:40.122357 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-2d2deb-openshift-default-79d58c4cbc-km8cv" Jul 19 08:32:40.225584 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:40.225547 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2d2deb-openshift-default-79d58c4cbc-km8cv" event={"ID":"02d3ea7e-ea37-4970-8afa-2040144cbb89","Type":"ContainerStarted","Data":"ea134ab394238f6bf2656df69201dd296b5549b8f5550cb1f3110f9d82db34ae"} Jul 19 08:32:40.486696 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:40.486658 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-2d2deb-openshift-default-79d58c4cbc-km8cv" Jul 19 08:32:40.488012 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:40.487986 2574 patch_prober.go:28] interesting pod/e2e-api-b-2d2deb-openshift-default-79d58c4cbc-km8cv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Jul 19 08:32:40.488151 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:40.488039 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2d2deb-openshift-default-79d58c4cbc-km8cv" podUID="02d3ea7e-ea37-4970-8afa-2040144cbb89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Jul 19 08:32:40.882900 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:40.882811 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 08:32:41.487799 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:41.487758 2574 patch_prober.go:28] interesting pod/e2e-api-b-2d2deb-openshift-default-79d58c4cbc-km8cv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Jul 19 08:32:41.488291 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:41.487820 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2d2deb-openshift-default-79d58c4cbc-km8cv" podUID="02d3ea7e-ea37-4970-8afa-2040144cbb89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Jul 19 08:32:42.487665 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:42.487622 2574 patch_prober.go:28] interesting pod/e2e-api-b-2d2deb-openshift-default-79d58c4cbc-km8cv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Jul 19 08:32:42.487862 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:42.487690 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2d2deb-openshift-default-79d58c4cbc-km8cv" podUID="02d3ea7e-ea37-4970-8afa-2040144cbb89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Jul 19 08:32:43.487458 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:43.487415 2574 patch_prober.go:28] interesting pod/e2e-api-b-2d2deb-openshift-default-79d58c4cbc-km8cv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Jul 19 08:32:43.487696 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:43.487504 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2d2deb-openshift-default-79d58c4cbc-km8cv" podUID="02d3ea7e-ea37-4970-8afa-2040144cbb89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Jul 19 08:32:44.492125 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:44.492093 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-2d2deb-openshift-default-79d58c4cbc-km8cv" Jul 19 08:32:44.492556 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:44.492462 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-2d2deb-openshift-default-79d58c4cbc-km8cv" Jul 19 08:32:44.493342 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:44.493315 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-2d2deb-openshift-default-79d58c4cbc-km8cv" Jul 19 08:32:47.692551 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:47.692500 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:32:47.794737 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:47.794696 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:32:49.173211 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:49.173172 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:32:49.274964 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:49.274930 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:32:49.410112 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:49.410070 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5da88c-667c58dfd6-74j86"] Jul 19 08:32:49.410422 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:49.410387 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5da88c-667c58dfd6-74j86" podUID="9619b8aa-97a1-4901-96db-302e4502d2e6" containerName="maas-api" containerID="cri-o://8e8135d88a71183df81813d7e962b4f37492e344bd612cd85cecb99fcbca09d5" gracePeriod=30 Jul 19 08:32:49.442718 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:49.442687 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:32:49.565176 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:49.565133 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:32:49.666107 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:49.666075 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:32:49.675514 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:49.675491 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5da88c-667c58dfd6-74j86" Jul 19 08:32:49.825994 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:49.825955 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9619b8aa-97a1-4901-96db-302e4502d2e6-openshift-service-ca\") pod \"9619b8aa-97a1-4901-96db-302e4502d2e6\" (UID: \"9619b8aa-97a1-4901-96db-302e4502d2e6\") " Jul 19 08:32:49.825994 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:49.826001 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tv5cd\" (UniqueName: \"kubernetes.io/projected/9619b8aa-97a1-4901-96db-302e4502d2e6-kube-api-access-tv5cd\") pod \"9619b8aa-97a1-4901-96db-302e4502d2e6\" (UID: \"9619b8aa-97a1-4901-96db-302e4502d2e6\") " Jul 19 08:32:49.826285 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:49.826055 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9619b8aa-97a1-4901-96db-302e4502d2e6-maas-api-tls\") pod \"9619b8aa-97a1-4901-96db-302e4502d2e6\" (UID: \"9619b8aa-97a1-4901-96db-302e4502d2e6\") " Jul 19 08:32:49.826432 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:49.826385 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9619b8aa-97a1-4901-96db-302e4502d2e6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9619b8aa-97a1-4901-96db-302e4502d2e6" (UID: "9619b8aa-97a1-4901-96db-302e4502d2e6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 08:32:49.828285 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:49.828255 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9619b8aa-97a1-4901-96db-302e4502d2e6-kube-api-access-tv5cd" (OuterVolumeSpecName: "kube-api-access-tv5cd") pod "9619b8aa-97a1-4901-96db-302e4502d2e6" (UID: "9619b8aa-97a1-4901-96db-302e4502d2e6"). InnerVolumeSpecName "kube-api-access-tv5cd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 08:32:49.828285 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:49.828268 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9619b8aa-97a1-4901-96db-302e4502d2e6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9619b8aa-97a1-4901-96db-302e4502d2e6" (UID: "9619b8aa-97a1-4901-96db-302e4502d2e6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 08:32:49.926919 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:49.926880 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9619b8aa-97a1-4901-96db-302e4502d2e6-openshift-service-ca\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:32:49.926919 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:49.926912 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tv5cd\" (UniqueName: \"kubernetes.io/projected/9619b8aa-97a1-4901-96db-302e4502d2e6-kube-api-access-tv5cd\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:32:49.926919 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:49.926923 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9619b8aa-97a1-4901-96db-302e4502d2e6-maas-api-tls\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:32:50.284675 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:50.284632 2574 generic.go:356] "Generic (PLEG): container finished" podID="9619b8aa-97a1-4901-96db-302e4502d2e6" containerID="8e8135d88a71183df81813d7e962b4f37492e344bd612cd85cecb99fcbca09d5" exitCode=0 Jul 19 08:32:50.285152 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:50.284684 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5da88c-667c58dfd6-74j86" Jul 19 08:32:50.285152 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:50.284716 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5da88c-667c58dfd6-74j86" event={"ID":"9619b8aa-97a1-4901-96db-302e4502d2e6","Type":"ContainerDied","Data":"8e8135d88a71183df81813d7e962b4f37492e344bd612cd85cecb99fcbca09d5"} Jul 19 08:32:50.285152 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:50.284774 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5da88c-667c58dfd6-74j86" event={"ID":"9619b8aa-97a1-4901-96db-302e4502d2e6","Type":"ContainerDied","Data":"55f6861e4c42e604e537c86ce3bd2bce830486724063a3e835a0c2d77337cb71"} Jul 19 08:32:50.285152 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:50.284799 2574 scope.go:117] "RemoveContainer" containerID="8e8135d88a71183df81813d7e962b4f37492e344bd612cd85cecb99fcbca09d5" Jul 19 08:32:50.294789 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:50.294760 2574 scope.go:117] "RemoveContainer" containerID="8e8135d88a71183df81813d7e962b4f37492e344bd612cd85cecb99fcbca09d5" Jul 19 08:32:50.295079 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:32:50.295047 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e8135d88a71183df81813d7e962b4f37492e344bd612cd85cecb99fcbca09d5\": container with ID starting with 8e8135d88a71183df81813d7e962b4f37492e344bd612cd85cecb99fcbca09d5 not found: ID does not exist" containerID="8e8135d88a71183df81813d7e962b4f37492e344bd612cd85cecb99fcbca09d5" Jul 19 08:32:50.295197 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:50.295087 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e8135d88a71183df81813d7e962b4f37492e344bd612cd85cecb99fcbca09d5"} err="failed to get container status \"8e8135d88a71183df81813d7e962b4f37492e344bd612cd85cecb99fcbca09d5\": rpc error: code = NotFound desc = could not find container \"8e8135d88a71183df81813d7e962b4f37492e344bd612cd85cecb99fcbca09d5\": container with ID starting with 8e8135d88a71183df81813d7e962b4f37492e344bd612cd85cecb99fcbca09d5 not found: ID does not exist" Jul 19 08:32:50.307279 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:50.307253 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5da88c-667c58dfd6-74j86"] Jul 19 08:32:50.310432 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:50.310406 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5da88c-667c58dfd6-74j86"] Jul 19 08:32:50.558857 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:50.558771 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9619b8aa-97a1-4901-96db-302e4502d2e6" path="/var/lib/kubelet/pods/9619b8aa-97a1-4901-96db-302e4502d2e6/volumes" Jul 19 08:32:53.168125 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:53.168093 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 08:32:53.595885 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:53.595836 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:32:59.211797 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:59.211752 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:32:59.271183 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:32:59.271117 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:33:04.217986 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:04.217947 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-5da88c-openshift-default-5c89fb85d5-s9flw"] Jul 19 08:33:04.218908 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:04.218871 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-5da88c-openshift-default-5c89fb85d5-s9flw" podUID="310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f" containerName="istio-proxy" containerID="cri-o://894a94f1b118ca19e07d363f0de7bb5a787564c123cc458748dfb68b611f8cba" gracePeriod=30 Jul 19 08:33:04.232022 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:04.231989 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:33:04.455890 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:04.455848 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:33:05.084929 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:05.084887 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:33:05.185511 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:05.185471 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:33:07.400838 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:07.400742 2574 patch_prober.go:28] interesting pod/e2e-api-a-5da88c-openshift-default-5c89fb85d5-s9flw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 19 08:33:07.400838 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:07.400804 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-5da88c-openshift-default-5c89fb85d5-s9flw" podUID="310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 19 08:33:09.385787 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:09.385752 2574 generic.go:356] "Generic (PLEG): container finished" podID="310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f" containerID="894a94f1b118ca19e07d363f0de7bb5a787564c123cc458748dfb68b611f8cba" exitCode=0 Jul 19 08:33:09.386237 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:09.385870 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5da88c-openshift-default-5c89fb85d5-s9flw" event={"ID":"310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f","Type":"ContainerDied","Data":"894a94f1b118ca19e07d363f0de7bb5a787564c123cc458748dfb68b611f8cba"} Jul 19 08:33:09.386237 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:09.385936 2574 scope.go:117] "RemoveContainer" containerID="cc1ba68e15fc8c56afbf0bcdde4e6699e5de322d39e74f49795ea38c9b4484f2" Jul 19 08:33:09.491813 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:09.491753 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5da88c-openshift-default-5c89fb85d5-s9flw" Jul 19 08:33:09.493673 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:09.493646 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f-credential-socket\") pod \"310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f\" (UID: \"310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f\") " Jul 19 08:33:09.493747 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:09.493695 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f-istiod-ca-cert\") pod \"310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f\" (UID: \"310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f\") " Jul 19 08:33:09.493747 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:09.493732 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f-workload-certs\") pod \"310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f\" (UID: \"310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f\") " Jul 19 08:33:09.493838 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:09.493753 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f-istio-envoy\") pod \"310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f\" (UID: \"310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f\") " Jul 19 08:33:09.493838 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:09.493794 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f-istio-data\") pod \"310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f\" (UID: \"310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f\") " Jul 19 08:33:09.493838 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:09.493825 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f-istio-token\") pod \"310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f\" (UID: \"310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f\") " Jul 19 08:33:09.494055 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:09.493881 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f-workload-socket\") pod \"310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f\" (UID: \"310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f\") " Jul 19 08:33:09.494055 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:09.493919 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hhv2t\" (UniqueName: \"kubernetes.io/projected/310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f-kube-api-access-hhv2t\") pod \"310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f\" (UID: \"310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f\") " Jul 19 08:33:09.494055 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:09.493944 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f-istio-podinfo\") pod \"310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f\" (UID: \"310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f\") " Jul 19 08:33:09.494055 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:09.493940 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f" (UID: "310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 08:33:09.494055 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:09.494013 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f" (UID: "310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 08:33:09.494316 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:09.494141 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f" (UID: "310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 08:33:09.494316 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:09.494145 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f" (UID: "310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 08:33:09.494316 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:09.494231 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f-workload-socket\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:33:09.494316 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:09.494250 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f-credential-socket\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:33:09.494316 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:09.494263 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f-istiod-ca-cert\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:33:09.494316 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:09.494271 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f-workload-certs\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:33:09.495101 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:09.494933 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f-istio-data" (OuterVolumeSpecName: "istio-data") pod "310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f" (UID: "310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 08:33:09.496091 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:09.496061 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f-kube-api-access-hhv2t" (OuterVolumeSpecName: "kube-api-access-hhv2t") pod "310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f" (UID: "310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f"). InnerVolumeSpecName "kube-api-access-hhv2t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 08:33:09.496274 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:09.496235 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f-istio-token" (OuterVolumeSpecName: "istio-token") pod "310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f" (UID: "310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 08:33:09.496274 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:09.496252 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f" (UID: "310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 19 08:33:09.496397 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:09.496278 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f" (UID: "310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 08:33:09.594840 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:09.594802 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f-istio-envoy\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:33:09.594840 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:09.594836 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f-istio-data\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:33:09.594840 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:09.594845 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f-istio-token\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:33:09.595098 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:09.594853 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhv2t\" (UniqueName: \"kubernetes.io/projected/310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f-kube-api-access-hhv2t\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:33:09.595098 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:09.594862 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f-istio-podinfo\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:33:10.391792 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:10.391757 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5da88c-openshift-default-5c89fb85d5-s9flw" Jul 19 08:33:10.392288 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:10.391791 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5da88c-openshift-default-5c89fb85d5-s9flw" event={"ID":"310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f","Type":"ContainerDied","Data":"20f2f32f0fdd63b763508032f7dc9d61c202f9d60bc396a40c5096a7dc1b3ce5"} Jul 19 08:33:10.392288 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:10.391845 2574 scope.go:117] "RemoveContainer" containerID="894a94f1b118ca19e07d363f0de7bb5a787564c123cc458748dfb68b611f8cba" Jul 19 08:33:10.413540 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:10.413497 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-5da88c-openshift-default-5c89fb85d5-s9flw"] Jul 19 08:33:10.420230 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:10.420190 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-5da88c-openshift-default-5c89fb85d5-s9flw"] Jul 19 08:33:10.558849 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:10.558815 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f" path="/var/lib/kubelet/pods/310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f/volumes" Jul 19 08:33:15.046304 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:15.046261 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2d2deb-6b8bf7f755-s99h9"] Jul 19 08:33:15.047089 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:15.047054 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2d2deb-6b8bf7f755-s99h9" podUID="001b4a01-654f-422f-bd4f-0e8350fdbedb" containerName="maas-api" containerID="cri-o://1ce000fbbc7e18fb7ad0e97b081723d61dfde9688343af96975737ac155598bf" gracePeriod=30 Jul 19 08:33:15.072593 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:15.072538 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:33:15.315071 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:15.315047 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2d2deb-6b8bf7f755-s99h9" Jul 19 08:33:15.339641 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:15.339608 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/001b4a01-654f-422f-bd4f-0e8350fdbedb-maas-api-tls\") pod \"001b4a01-654f-422f-bd4f-0e8350fdbedb\" (UID: \"001b4a01-654f-422f-bd4f-0e8350fdbedb\") " Jul 19 08:33:15.339808 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:15.339704 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/001b4a01-654f-422f-bd4f-0e8350fdbedb-openshift-service-ca\") pod \"001b4a01-654f-422f-bd4f-0e8350fdbedb\" (UID: \"001b4a01-654f-422f-bd4f-0e8350fdbedb\") " Jul 19 08:33:15.339808 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:15.339726 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h2x59\" (UniqueName: \"kubernetes.io/projected/001b4a01-654f-422f-bd4f-0e8350fdbedb-kube-api-access-h2x59\") pod \"001b4a01-654f-422f-bd4f-0e8350fdbedb\" (UID: \"001b4a01-654f-422f-bd4f-0e8350fdbedb\") " Jul 19 08:33:15.340135 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:15.340104 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/001b4a01-654f-422f-bd4f-0e8350fdbedb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "001b4a01-654f-422f-bd4f-0e8350fdbedb" (UID: "001b4a01-654f-422f-bd4f-0e8350fdbedb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 08:33:15.341871 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:15.341846 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/001b4a01-654f-422f-bd4f-0e8350fdbedb-kube-api-access-h2x59" (OuterVolumeSpecName: "kube-api-access-h2x59") pod "001b4a01-654f-422f-bd4f-0e8350fdbedb" (UID: "001b4a01-654f-422f-bd4f-0e8350fdbedb"). InnerVolumeSpecName "kube-api-access-h2x59". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 08:33:15.341996 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:15.341878 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/001b4a01-654f-422f-bd4f-0e8350fdbedb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "001b4a01-654f-422f-bd4f-0e8350fdbedb" (UID: "001b4a01-654f-422f-bd4f-0e8350fdbedb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 08:33:15.419367 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:15.419327 2574 generic.go:356] "Generic (PLEG): container finished" podID="001b4a01-654f-422f-bd4f-0e8350fdbedb" containerID="1ce000fbbc7e18fb7ad0e97b081723d61dfde9688343af96975737ac155598bf" exitCode=0 Jul 19 08:33:15.419564 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:15.419395 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2d2deb-6b8bf7f755-s99h9" Jul 19 08:33:15.419564 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:15.419419 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2d2deb-6b8bf7f755-s99h9" event={"ID":"001b4a01-654f-422f-bd4f-0e8350fdbedb","Type":"ContainerDied","Data":"1ce000fbbc7e18fb7ad0e97b081723d61dfde9688343af96975737ac155598bf"} Jul 19 08:33:15.419564 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:15.419556 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2d2deb-6b8bf7f755-s99h9" event={"ID":"001b4a01-654f-422f-bd4f-0e8350fdbedb","Type":"ContainerDied","Data":"2b31ad29b71a472889637ea9a02cc4f725d493233e3ef27aabe45d677b8290a7"} Jul 19 08:33:15.419722 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:15.419591 2574 scope.go:117] "RemoveContainer" containerID="1ce000fbbc7e18fb7ad0e97b081723d61dfde9688343af96975737ac155598bf" Jul 19 08:33:15.430898 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:15.430877 2574 scope.go:117] "RemoveContainer" containerID="1ce000fbbc7e18fb7ad0e97b081723d61dfde9688343af96975737ac155598bf" Jul 19 08:33:15.431161 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:33:15.431140 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ce000fbbc7e18fb7ad0e97b081723d61dfde9688343af96975737ac155598bf\": container with ID starting with 1ce000fbbc7e18fb7ad0e97b081723d61dfde9688343af96975737ac155598bf not found: ID does not exist" containerID="1ce000fbbc7e18fb7ad0e97b081723d61dfde9688343af96975737ac155598bf" Jul 19 08:33:15.431216 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:15.431170 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ce000fbbc7e18fb7ad0e97b081723d61dfde9688343af96975737ac155598bf"} err="failed to get container status \"1ce000fbbc7e18fb7ad0e97b081723d61dfde9688343af96975737ac155598bf\": rpc error: code = NotFound desc = could not find container \"1ce000fbbc7e18fb7ad0e97b081723d61dfde9688343af96975737ac155598bf\": container with ID starting with 1ce000fbbc7e18fb7ad0e97b081723d61dfde9688343af96975737ac155598bf not found: ID does not exist" Jul 19 08:33:15.440401 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:15.440374 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/001b4a01-654f-422f-bd4f-0e8350fdbedb-openshift-service-ca\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:33:15.440401 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:15.440402 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2x59\" (UniqueName: \"kubernetes.io/projected/001b4a01-654f-422f-bd4f-0e8350fdbedb-kube-api-access-h2x59\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:33:15.440584 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:15.440413 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/001b4a01-654f-422f-bd4f-0e8350fdbedb-maas-api-tls\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:33:15.442094 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:15.442074 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2d2deb-6b8bf7f755-s99h9"] Jul 19 08:33:15.445449 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:15.445429 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2d2deb-6b8bf7f755-s99h9"] Jul 19 08:33:16.559181 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:16.559148 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="001b4a01-654f-422f-bd4f-0e8350fdbedb" path="/var/lib/kubelet/pods/001b4a01-654f-422f-bd4f-0e8350fdbedb/volumes" Jul 19 08:33:18.569879 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:18.569847 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 08:33:19.663491 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:19.663447 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:33:24.768639 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:24.768585 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:33:25.281535 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:25.281491 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:33:25.337867 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:25.337833 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:33:30.277024 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:30.276981 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-2d2deb-openshift-default-79d58c4cbc-km8cv"] Jul 19 08:33:30.277537 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:30.277219 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-2d2deb-openshift-default-79d58c4cbc-km8cv" podUID="02d3ea7e-ea37-4970-8afa-2040144cbb89" containerName="istio-proxy" containerID="cri-o://ea134ab394238f6bf2656df69201dd296b5549b8f5550cb1f3110f9d82db34ae" gracePeriod=30 Jul 19 08:33:30.299189 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:30.299152 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:33:30.518446 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:30.518408 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:33:30.840153 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:30.840110 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:33:31.112168 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:31.112082 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-72ce60-openshift-default-7fdc94f55-24snn"] Jul 19 08:33:31.112647 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:31.112623 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="001b4a01-654f-422f-bd4f-0e8350fdbedb" containerName="maas-api" Jul 19 08:33:31.112647 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:31.112646 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="001b4a01-654f-422f-bd4f-0e8350fdbedb" containerName="maas-api" Jul 19 08:33:31.112847 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:31.112672 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f" containerName="istio-proxy" Jul 19 08:33:31.112847 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:31.112680 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f" containerName="istio-proxy" Jul 19 08:33:31.112847 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:31.112691 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f" containerName="istio-proxy" Jul 19 08:33:31.112847 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:31.112699 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f" containerName="istio-proxy" Jul 19 08:33:31.112847 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:31.112724 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9619b8aa-97a1-4901-96db-302e4502d2e6" containerName="maas-api" Jul 19 08:33:31.112847 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:31.112732 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="9619b8aa-97a1-4901-96db-302e4502d2e6" containerName="maas-api" Jul 19 08:33:31.112847 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:31.112836 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f" containerName="istio-proxy" Jul 19 08:33:31.113178 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:31.112851 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="9619b8aa-97a1-4901-96db-302e4502d2e6" containerName="maas-api" Jul 19 08:33:31.113178 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:31.112861 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="001b4a01-654f-422f-bd4f-0e8350fdbedb" containerName="maas-api" Jul 19 08:33:31.113178 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:31.112873 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="310ae9a9-00fe-404c-a0f6-2b1e0ef14d3f" containerName="istio-proxy" Jul 19 08:33:31.116028 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:31.116003 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-72ce60-openshift-default-7fdc94f55-24snn" Jul 19 08:33:31.118636 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:31.118609 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-72ce60-openshift-default-dockercfg-hcmlp\"" Jul 19 08:33:31.126769 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:31.126740 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-72ce60-openshift-default-7fdc94f55-24snn"] Jul 19 08:33:31.140251 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:31.140222 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:33:31.177023 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:31.176988 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c3b1d54-8921-4761-b776-182f6c7889b1-workload-socket\") pod \"e2e-shared-a-72ce60-openshift-default-7fdc94f55-24snn\" (UID: \"6c3b1d54-8921-4761-b776-182f6c7889b1\") " pod="openshift-ingress/e2e-shared-a-72ce60-openshift-default-7fdc94f55-24snn" Jul 19 08:33:31.177023 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:31.177031 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2lbg\" (UniqueName: \"kubernetes.io/projected/6c3b1d54-8921-4761-b776-182f6c7889b1-kube-api-access-z2lbg\") pod \"e2e-shared-a-72ce60-openshift-default-7fdc94f55-24snn\" (UID: \"6c3b1d54-8921-4761-b776-182f6c7889b1\") " pod="openshift-ingress/e2e-shared-a-72ce60-openshift-default-7fdc94f55-24snn" Jul 19 08:33:31.177260 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:31.177063 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c3b1d54-8921-4761-b776-182f6c7889b1-istio-token\") pod \"e2e-shared-a-72ce60-openshift-default-7fdc94f55-24snn\" (UID: \"6c3b1d54-8921-4761-b776-182f6c7889b1\") " pod="openshift-ingress/e2e-shared-a-72ce60-openshift-default-7fdc94f55-24snn" Jul 19 08:33:31.177260 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:31.177146 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c3b1d54-8921-4761-b776-182f6c7889b1-workload-certs\") pod \"e2e-shared-a-72ce60-openshift-default-7fdc94f55-24snn\" (UID: \"6c3b1d54-8921-4761-b776-182f6c7889b1\") " pod="openshift-ingress/e2e-shared-a-72ce60-openshift-default-7fdc94f55-24snn" Jul 19 08:33:31.177260 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:31.177194 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c3b1d54-8921-4761-b776-182f6c7889b1-credential-socket\") pod \"e2e-shared-a-72ce60-openshift-default-7fdc94f55-24snn\" (UID: \"6c3b1d54-8921-4761-b776-182f6c7889b1\") " pod="openshift-ingress/e2e-shared-a-72ce60-openshift-default-7fdc94f55-24snn" Jul 19 08:33:31.177260 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:31.177232 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c3b1d54-8921-4761-b776-182f6c7889b1-istio-envoy\") pod \"e2e-shared-a-72ce60-openshift-default-7fdc94f55-24snn\" (UID: \"6c3b1d54-8921-4761-b776-182f6c7889b1\") " pod="openshift-ingress/e2e-shared-a-72ce60-openshift-default-7fdc94f55-24snn" Jul 19 08:33:31.177426 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:31.177262 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c3b1d54-8921-4761-b776-182f6c7889b1-istio-podinfo\") pod \"e2e-shared-a-72ce60-openshift-default-7fdc94f55-24snn\" (UID: \"6c3b1d54-8921-4761-b776-182f6c7889b1\") " pod="openshift-ingress/e2e-shared-a-72ce60-openshift-default-7fdc94f55-24snn" Jul 19 08:33:31.177426 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:31.177291 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c3b1d54-8921-4761-b776-182f6c7889b1-istiod-ca-cert\") pod \"e2e-shared-a-72ce60-openshift-default-7fdc94f55-24snn\" (UID: \"6c3b1d54-8921-4761-b776-182f6c7889b1\") " pod="openshift-ingress/e2e-shared-a-72ce60-openshift-default-7fdc94f55-24snn" Jul 19 08:33:31.177426 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:31.177329 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c3b1d54-8921-4761-b776-182f6c7889b1-istio-data\") pod \"e2e-shared-a-72ce60-openshift-default-7fdc94f55-24snn\" (UID: \"6c3b1d54-8921-4761-b776-182f6c7889b1\") " pod="openshift-ingress/e2e-shared-a-72ce60-openshift-default-7fdc94f55-24snn" Jul 19 08:33:31.278374 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:31.278339 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c3b1d54-8921-4761-b776-182f6c7889b1-istio-token\") pod \"e2e-shared-a-72ce60-openshift-default-7fdc94f55-24snn\" (UID: \"6c3b1d54-8921-4761-b776-182f6c7889b1\") " pod="openshift-ingress/e2e-shared-a-72ce60-openshift-default-7fdc94f55-24snn" Jul 19 08:33:31.278374 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:31.278389 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c3b1d54-8921-4761-b776-182f6c7889b1-workload-certs\") pod \"e2e-shared-a-72ce60-openshift-default-7fdc94f55-24snn\" (UID: \"6c3b1d54-8921-4761-b776-182f6c7889b1\") " pod="openshift-ingress/e2e-shared-a-72ce60-openshift-default-7fdc94f55-24snn" Jul 19 08:33:31.278985 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:31.278514 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c3b1d54-8921-4761-b776-182f6c7889b1-credential-socket\") pod \"e2e-shared-a-72ce60-openshift-default-7fdc94f55-24snn\" (UID: \"6c3b1d54-8921-4761-b776-182f6c7889b1\") " pod="openshift-ingress/e2e-shared-a-72ce60-openshift-default-7fdc94f55-24snn" Jul 19 08:33:31.278985 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:31.278602 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c3b1d54-8921-4761-b776-182f6c7889b1-istio-envoy\") pod \"e2e-shared-a-72ce60-openshift-default-7fdc94f55-24snn\" (UID: \"6c3b1d54-8921-4761-b776-182f6c7889b1\") " pod="openshift-ingress/e2e-shared-a-72ce60-openshift-default-7fdc94f55-24snn" Jul 19 08:33:31.278985 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:31.278636 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c3b1d54-8921-4761-b776-182f6c7889b1-istio-podinfo\") pod \"e2e-shared-a-72ce60-openshift-default-7fdc94f55-24snn\" (UID: \"6c3b1d54-8921-4761-b776-182f6c7889b1\") " pod="openshift-ingress/e2e-shared-a-72ce60-openshift-default-7fdc94f55-24snn" Jul 19 08:33:31.278985 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:31.278667 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c3b1d54-8921-4761-b776-182f6c7889b1-istiod-ca-cert\") pod \"e2e-shared-a-72ce60-openshift-default-7fdc94f55-24snn\" (UID: \"6c3b1d54-8921-4761-b776-182f6c7889b1\") " pod="openshift-ingress/e2e-shared-a-72ce60-openshift-default-7fdc94f55-24snn" Jul 19 08:33:31.278985 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:31.278714 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c3b1d54-8921-4761-b776-182f6c7889b1-istio-data\") pod \"e2e-shared-a-72ce60-openshift-default-7fdc94f55-24snn\" (UID: \"6c3b1d54-8921-4761-b776-182f6c7889b1\") " pod="openshift-ingress/e2e-shared-a-72ce60-openshift-default-7fdc94f55-24snn" Jul 19 08:33:31.278985 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:31.278779 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c3b1d54-8921-4761-b776-182f6c7889b1-workload-certs\") pod \"e2e-shared-a-72ce60-openshift-default-7fdc94f55-24snn\" (UID: \"6c3b1d54-8921-4761-b776-182f6c7889b1\") " pod="openshift-ingress/e2e-shared-a-72ce60-openshift-default-7fdc94f55-24snn" Jul 19 08:33:31.278985 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:31.278811 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c3b1d54-8921-4761-b776-182f6c7889b1-workload-socket\") pod \"e2e-shared-a-72ce60-openshift-default-7fdc94f55-24snn\" (UID: \"6c3b1d54-8921-4761-b776-182f6c7889b1\") " pod="openshift-ingress/e2e-shared-a-72ce60-openshift-default-7fdc94f55-24snn" Jul 19 08:33:31.278985 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:31.278840 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2lbg\" (UniqueName: \"kubernetes.io/projected/6c3b1d54-8921-4761-b776-182f6c7889b1-kube-api-access-z2lbg\") pod \"e2e-shared-a-72ce60-openshift-default-7fdc94f55-24snn\" (UID: \"6c3b1d54-8921-4761-b776-182f6c7889b1\") " pod="openshift-ingress/e2e-shared-a-72ce60-openshift-default-7fdc94f55-24snn" Jul 19 08:33:31.278985 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:31.278868 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c3b1d54-8921-4761-b776-182f6c7889b1-credential-socket\") pod \"e2e-shared-a-72ce60-openshift-default-7fdc94f55-24snn\" (UID: \"6c3b1d54-8921-4761-b776-182f6c7889b1\") " pod="openshift-ingress/e2e-shared-a-72ce60-openshift-default-7fdc94f55-24snn" Jul 19 08:33:31.279404 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:31.279007 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c3b1d54-8921-4761-b776-182f6c7889b1-istio-data\") pod \"e2e-shared-a-72ce60-openshift-default-7fdc94f55-24snn\" (UID: \"6c3b1d54-8921-4761-b776-182f6c7889b1\") " pod="openshift-ingress/e2e-shared-a-72ce60-openshift-default-7fdc94f55-24snn" Jul 19 08:33:31.279404 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:31.279218 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c3b1d54-8921-4761-b776-182f6c7889b1-workload-socket\") pod \"e2e-shared-a-72ce60-openshift-default-7fdc94f55-24snn\" (UID: \"6c3b1d54-8921-4761-b776-182f6c7889b1\") " pod="openshift-ingress/e2e-shared-a-72ce60-openshift-default-7fdc94f55-24snn" Jul 19 08:33:31.279404 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:31.279375 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c3b1d54-8921-4761-b776-182f6c7889b1-istiod-ca-cert\") pod \"e2e-shared-a-72ce60-openshift-default-7fdc94f55-24snn\" (UID: \"6c3b1d54-8921-4761-b776-182f6c7889b1\") " pod="openshift-ingress/e2e-shared-a-72ce60-openshift-default-7fdc94f55-24snn" Jul 19 08:33:31.280994 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:31.280972 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c3b1d54-8921-4761-b776-182f6c7889b1-istio-envoy\") pod \"e2e-shared-a-72ce60-openshift-default-7fdc94f55-24snn\" (UID: \"6c3b1d54-8921-4761-b776-182f6c7889b1\") " pod="openshift-ingress/e2e-shared-a-72ce60-openshift-default-7fdc94f55-24snn" Jul 19 08:33:31.281228 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:31.281207 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c3b1d54-8921-4761-b776-182f6c7889b1-istio-podinfo\") pod \"e2e-shared-a-72ce60-openshift-default-7fdc94f55-24snn\" (UID: \"6c3b1d54-8921-4761-b776-182f6c7889b1\") " pod="openshift-ingress/e2e-shared-a-72ce60-openshift-default-7fdc94f55-24snn" Jul 19 08:33:31.286400 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:31.286374 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c3b1d54-8921-4761-b776-182f6c7889b1-istio-token\") pod \"e2e-shared-a-72ce60-openshift-default-7fdc94f55-24snn\" (UID: \"6c3b1d54-8921-4761-b776-182f6c7889b1\") " pod="openshift-ingress/e2e-shared-a-72ce60-openshift-default-7fdc94f55-24snn" Jul 19 08:33:31.286511 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:31.286492 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2lbg\" (UniqueName: \"kubernetes.io/projected/6c3b1d54-8921-4761-b776-182f6c7889b1-kube-api-access-z2lbg\") pod \"e2e-shared-a-72ce60-openshift-default-7fdc94f55-24snn\" (UID: \"6c3b1d54-8921-4761-b776-182f6c7889b1\") " pod="openshift-ingress/e2e-shared-a-72ce60-openshift-default-7fdc94f55-24snn" Jul 19 08:33:31.301745 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:31.301708 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:33:31.429988 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:31.429892 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-72ce60-openshift-default-7fdc94f55-24snn" Jul 19 08:33:31.575133 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:31.575107 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-72ce60-openshift-default-7fdc94f55-24snn"] Jul 19 08:33:31.577160 ip-10-0-136-70 kubenswrapper[2574]: W0719 08:33:31.577125 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c3b1d54_8921_4761_b776_182f6c7889b1.slice/crio-56cd13d0a820494ac2ad565097636f7a7b64411153c361022675a116ae61f288 WatchSource:0}: Error finding container 56cd13d0a820494ac2ad565097636f7a7b64411153c361022675a116ae61f288: Status 404 returned error can't find the container with id 56cd13d0a820494ac2ad565097636f7a7b64411153c361022675a116ae61f288 Jul 19 08:33:31.579173 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:31.579140 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 19 08:33:31.579264 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:31.579221 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 19 08:33:31.579321 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:31.579263 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 19 08:33:32.508823 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:32.508788 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-72ce60-openshift-default-7fdc94f55-24snn" event={"ID":"6c3b1d54-8921-4761-b776-182f6c7889b1","Type":"ContainerStarted","Data":"66a22f07cd26556bb17a657578b5fb7f2ac38eeed2323099571e4ffb271caf7f"} Jul 19 08:33:32.509222 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:32.508846 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-72ce60-openshift-default-7fdc94f55-24snn" event={"ID":"6c3b1d54-8921-4761-b776-182f6c7889b1","Type":"ContainerStarted","Data":"56cd13d0a820494ac2ad565097636f7a7b64411153c361022675a116ae61f288"} Jul 19 08:33:32.530280 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:32.530228 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-72ce60-openshift-default-7fdc94f55-24snn" podStartSLOduration=1.530213291 podStartE2EDuration="1.530213291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 08:33:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:33:32.528598788 +0000 UTC m=+3078.882908390" watchObservedRunningTime="2026-07-19 08:33:32.530213291 +0000 UTC m=+3078.884522891" Jul 19 08:33:33.430253 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:33.430211 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-72ce60-openshift-default-7fdc94f55-24snn" Jul 19 08:33:33.435275 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:33.435244 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-72ce60-openshift-default-7fdc94f55-24snn" Jul 19 08:33:33.513899 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:33.513860 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-72ce60-openshift-default-7fdc94f55-24snn" Jul 19 08:33:33.514939 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:33.514919 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-72ce60-openshift-default-7fdc94f55-24snn" Jul 19 08:33:34.412045 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:34.412007 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:33:34.511135 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:34.511089 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:33:34.635417 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:34.635384 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-72ce60-56fc9c7fb7-xfqrk"] Jul 19 08:33:34.639623 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:34.639599 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-72ce60-56fc9c7fb7-xfqrk" Jul 19 08:33:34.642457 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:34.642424 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-72ce60-serving-cert\"" Jul 19 08:33:34.642662 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:34.642493 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 19 08:33:34.642746 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:34.642502 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-tdbkh\"" Jul 19 08:33:34.642746 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:34.642597 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 19 08:33:34.649766 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:34.649276 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-72ce60-56fc9c7fb7-xfqrk"] Jul 19 08:33:34.682324 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:34.682201 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:33:34.714162 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:34.714126 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmmpz\" (UniqueName: \"kubernetes.io/projected/bcd38542-3ecd-4d77-a6dd-af4503096eca-kube-api-access-qmmpz\") pod \"maas-api-e2e-shared-a-72ce60-56fc9c7fb7-xfqrk\" (UID: \"bcd38542-3ecd-4d77-a6dd-af4503096eca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-72ce60-56fc9c7fb7-xfqrk" Jul 19 08:33:34.714361 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:34.714180 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcd38542-3ecd-4d77-a6dd-af4503096eca-openshift-service-ca\") pod \"maas-api-e2e-shared-a-72ce60-56fc9c7fb7-xfqrk\" (UID: \"bcd38542-3ecd-4d77-a6dd-af4503096eca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-72ce60-56fc9c7fb7-xfqrk" Jul 19 08:33:34.714361 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:34.714251 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcd38542-3ecd-4d77-a6dd-af4503096eca-maas-api-tls\") pod \"maas-api-e2e-shared-a-72ce60-56fc9c7fb7-xfqrk\" (UID: \"bcd38542-3ecd-4d77-a6dd-af4503096eca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-72ce60-56fc9c7fb7-xfqrk" Jul 19 08:33:34.815458 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:34.815419 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmmpz\" (UniqueName: \"kubernetes.io/projected/bcd38542-3ecd-4d77-a6dd-af4503096eca-kube-api-access-qmmpz\") pod \"maas-api-e2e-shared-a-72ce60-56fc9c7fb7-xfqrk\" (UID: \"bcd38542-3ecd-4d77-a6dd-af4503096eca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-72ce60-56fc9c7fb7-xfqrk" Jul 19 08:33:34.815717 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:34.815476 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcd38542-3ecd-4d77-a6dd-af4503096eca-openshift-service-ca\") pod \"maas-api-e2e-shared-a-72ce60-56fc9c7fb7-xfqrk\" (UID: \"bcd38542-3ecd-4d77-a6dd-af4503096eca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-72ce60-56fc9c7fb7-xfqrk" Jul 19 08:33:34.815717 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:34.815548 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcd38542-3ecd-4d77-a6dd-af4503096eca-maas-api-tls\") pod \"maas-api-e2e-shared-a-72ce60-56fc9c7fb7-xfqrk\" (UID: \"bcd38542-3ecd-4d77-a6dd-af4503096eca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-72ce60-56fc9c7fb7-xfqrk" Jul 19 08:33:34.815717 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:33:34.815658 2574 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-72ce60-serving-cert: secret "maas-api-e2e-shared-a-72ce60-serving-cert" not found Jul 19 08:33:34.815717 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:33:34.815718 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bcd38542-3ecd-4d77-a6dd-af4503096eca-maas-api-tls podName:bcd38542-3ecd-4d77-a6dd-af4503096eca nodeName:}" failed. No retries permitted until 2026-07-19 08:33:35.31570211 +0000 UTC m=+3081.670011692 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bcd38542-3ecd-4d77-a6dd-af4503096eca-maas-api-tls") pod "maas-api-e2e-shared-a-72ce60-56fc9c7fb7-xfqrk" (UID: "bcd38542-3ecd-4d77-a6dd-af4503096eca") : secret "maas-api-e2e-shared-a-72ce60-serving-cert" not found Jul 19 08:33:34.816149 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:34.816127 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcd38542-3ecd-4d77-a6dd-af4503096eca-openshift-service-ca\") pod \"maas-api-e2e-shared-a-72ce60-56fc9c7fb7-xfqrk\" (UID: \"bcd38542-3ecd-4d77-a6dd-af4503096eca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-72ce60-56fc9c7fb7-xfqrk" Jul 19 08:33:34.824004 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:34.823964 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmmpz\" (UniqueName: \"kubernetes.io/projected/bcd38542-3ecd-4d77-a6dd-af4503096eca-kube-api-access-qmmpz\") pod \"maas-api-e2e-shared-a-72ce60-56fc9c7fb7-xfqrk\" (UID: \"bcd38542-3ecd-4d77-a6dd-af4503096eca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-72ce60-56fc9c7fb7-xfqrk" Jul 19 08:33:35.055489 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:35.055451 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:33:35.156343 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:35.156310 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:33:35.256465 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:35.256425 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:33:35.321985 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:35.321900 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcd38542-3ecd-4d77-a6dd-af4503096eca-maas-api-tls\") pod \"maas-api-e2e-shared-a-72ce60-56fc9c7fb7-xfqrk\" (UID: \"bcd38542-3ecd-4d77-a6dd-af4503096eca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-72ce60-56fc9c7fb7-xfqrk" Jul 19 08:33:35.324731 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:35.324701 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcd38542-3ecd-4d77-a6dd-af4503096eca-maas-api-tls\") pod \"maas-api-e2e-shared-a-72ce60-56fc9c7fb7-xfqrk\" (UID: \"bcd38542-3ecd-4d77-a6dd-af4503096eca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-72ce60-56fc9c7fb7-xfqrk" Jul 19 08:33:35.528443 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:35.528414 2574 generic.go:356] "Generic (PLEG): container finished" podID="02d3ea7e-ea37-4970-8afa-2040144cbb89" containerID="ea134ab394238f6bf2656df69201dd296b5549b8f5550cb1f3110f9d82db34ae" exitCode=0 Jul 19 08:33:35.528582 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:35.528492 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2d2deb-openshift-default-79d58c4cbc-km8cv" event={"ID":"02d3ea7e-ea37-4970-8afa-2040144cbb89","Type":"ContainerDied","Data":"ea134ab394238f6bf2656df69201dd296b5549b8f5550cb1f3110f9d82db34ae"} Jul 19 08:33:35.528582 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:35.528574 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2d2deb-openshift-default-79d58c4cbc-km8cv" event={"ID":"02d3ea7e-ea37-4970-8afa-2040144cbb89","Type":"ContainerDied","Data":"4cc7e42721f9d2c012e8a07854e2c2393a8fba1dcde29fb02b0f24eb27f88c8a"} Jul 19 08:33:35.528679 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:35.528593 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4cc7e42721f9d2c012e8a07854e2c2393a8fba1dcde29fb02b0f24eb27f88c8a" Jul 19 08:33:35.528679 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:35.528608 2574 scope.go:117] "RemoveContainer" containerID="175049f9d0eef6215d328c757144a6b0867f9f0e349b94982112030ed167a0a9" Jul 19 08:33:35.548800 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:35.548780 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2d2deb-openshift-default-79d58c4cbc-km8cv" Jul 19 08:33:35.556186 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:35.556165 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-72ce60-56fc9c7fb7-xfqrk" Jul 19 08:33:35.625730 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:35.624802 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02d3ea7e-ea37-4970-8afa-2040144cbb89-workload-certs\") pod \"02d3ea7e-ea37-4970-8afa-2040144cbb89\" (UID: \"02d3ea7e-ea37-4970-8afa-2040144cbb89\") " Jul 19 08:33:35.625730 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:35.624861 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02d3ea7e-ea37-4970-8afa-2040144cbb89-istio-envoy\") pod \"02d3ea7e-ea37-4970-8afa-2040144cbb89\" (UID: \"02d3ea7e-ea37-4970-8afa-2040144cbb89\") " Jul 19 08:33:35.625730 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:35.624944 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k52jn\" (UniqueName: \"kubernetes.io/projected/02d3ea7e-ea37-4970-8afa-2040144cbb89-kube-api-access-k52jn\") pod \"02d3ea7e-ea37-4970-8afa-2040144cbb89\" (UID: \"02d3ea7e-ea37-4970-8afa-2040144cbb89\") " Jul 19 08:33:35.625730 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:35.624973 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02d3ea7e-ea37-4970-8afa-2040144cbb89-istio-data\") pod \"02d3ea7e-ea37-4970-8afa-2040144cbb89\" (UID: \"02d3ea7e-ea37-4970-8afa-2040144cbb89\") " Jul 19 08:33:35.625730 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:35.625010 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02d3ea7e-ea37-4970-8afa-2040144cbb89-istiod-ca-cert\") pod \"02d3ea7e-ea37-4970-8afa-2040144cbb89\" (UID: \"02d3ea7e-ea37-4970-8afa-2040144cbb89\") " Jul 19 08:33:35.625730 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:35.625048 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02d3ea7e-ea37-4970-8afa-2040144cbb89-credential-socket\") pod \"02d3ea7e-ea37-4970-8afa-2040144cbb89\" (UID: \"02d3ea7e-ea37-4970-8afa-2040144cbb89\") " Jul 19 08:33:35.625730 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:35.625080 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02d3ea7e-ea37-4970-8afa-2040144cbb89-istio-podinfo\") pod \"02d3ea7e-ea37-4970-8afa-2040144cbb89\" (UID: \"02d3ea7e-ea37-4970-8afa-2040144cbb89\") " Jul 19 08:33:35.625730 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:35.625115 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02d3ea7e-ea37-4970-8afa-2040144cbb89-workload-socket\") pod \"02d3ea7e-ea37-4970-8afa-2040144cbb89\" (UID: \"02d3ea7e-ea37-4970-8afa-2040144cbb89\") " Jul 19 08:33:35.625730 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:35.625150 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02d3ea7e-ea37-4970-8afa-2040144cbb89-istio-token\") pod \"02d3ea7e-ea37-4970-8afa-2040144cbb89\" (UID: \"02d3ea7e-ea37-4970-8afa-2040144cbb89\") " Jul 19 08:33:35.625730 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:35.625306 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02d3ea7e-ea37-4970-8afa-2040144cbb89-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "02d3ea7e-ea37-4970-8afa-2040144cbb89" (UID: "02d3ea7e-ea37-4970-8afa-2040144cbb89"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 08:33:35.625730 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:35.625669 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02d3ea7e-ea37-4970-8afa-2040144cbb89-workload-certs\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:33:35.626544 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:35.626435 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02d3ea7e-ea37-4970-8afa-2040144cbb89-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "02d3ea7e-ea37-4970-8afa-2040144cbb89" (UID: "02d3ea7e-ea37-4970-8afa-2040144cbb89"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 08:33:35.626544 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:35.626478 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02d3ea7e-ea37-4970-8afa-2040144cbb89-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "02d3ea7e-ea37-4970-8afa-2040144cbb89" (UID: "02d3ea7e-ea37-4970-8afa-2040144cbb89"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 08:33:35.627057 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:35.626799 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02d3ea7e-ea37-4970-8afa-2040144cbb89-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "02d3ea7e-ea37-4970-8afa-2040144cbb89" (UID: "02d3ea7e-ea37-4970-8afa-2040144cbb89"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 08:33:35.627634 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:35.627611 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02d3ea7e-ea37-4970-8afa-2040144cbb89-istio-data" (OuterVolumeSpecName: "istio-data") pod "02d3ea7e-ea37-4970-8afa-2040144cbb89" (UID: "02d3ea7e-ea37-4970-8afa-2040144cbb89"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 08:33:35.628122 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:35.628095 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02d3ea7e-ea37-4970-8afa-2040144cbb89-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "02d3ea7e-ea37-4970-8afa-2040144cbb89" (UID: "02d3ea7e-ea37-4970-8afa-2040144cbb89"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 08:33:35.628235 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:35.628213 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02d3ea7e-ea37-4970-8afa-2040144cbb89-kube-api-access-k52jn" (OuterVolumeSpecName: "kube-api-access-k52jn") pod "02d3ea7e-ea37-4970-8afa-2040144cbb89" (UID: "02d3ea7e-ea37-4970-8afa-2040144cbb89"). InnerVolumeSpecName "kube-api-access-k52jn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 08:33:35.628456 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:35.628437 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02d3ea7e-ea37-4970-8afa-2040144cbb89-istio-token" (OuterVolumeSpecName: "istio-token") pod "02d3ea7e-ea37-4970-8afa-2040144cbb89" (UID: "02d3ea7e-ea37-4970-8afa-2040144cbb89"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 08:33:35.629168 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:35.629140 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/02d3ea7e-ea37-4970-8afa-2040144cbb89-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "02d3ea7e-ea37-4970-8afa-2040144cbb89" (UID: "02d3ea7e-ea37-4970-8afa-2040144cbb89"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 19 08:33:35.697901 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:35.697872 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-72ce60-56fc9c7fb7-xfqrk"] Jul 19 08:33:35.699772 ip-10-0-136-70 kubenswrapper[2574]: W0719 08:33:35.699740 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcd38542_3ecd_4d77_a6dd_af4503096eca.slice/crio-3259f1b10ed003151440d5c03ee7311fbda8eb0e21ab530812fdac98e10c1497 WatchSource:0}: Error finding container 3259f1b10ed003151440d5c03ee7311fbda8eb0e21ab530812fdac98e10c1497: Status 404 returned error can't find the container with id 3259f1b10ed003151440d5c03ee7311fbda8eb0e21ab530812fdac98e10c1497 Jul 19 08:33:35.726336 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:35.726284 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02d3ea7e-ea37-4970-8afa-2040144cbb89-credential-socket\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:33:35.726336 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:35.726333 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02d3ea7e-ea37-4970-8afa-2040144cbb89-istio-podinfo\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:33:35.726336 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:35.726344 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02d3ea7e-ea37-4970-8afa-2040144cbb89-workload-socket\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:33:35.726712 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:35.726362 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02d3ea7e-ea37-4970-8afa-2040144cbb89-istio-token\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:33:35.726712 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:35.726373 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02d3ea7e-ea37-4970-8afa-2040144cbb89-istio-envoy\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:33:35.726712 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:35.726396 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k52jn\" (UniqueName: \"kubernetes.io/projected/02d3ea7e-ea37-4970-8afa-2040144cbb89-kube-api-access-k52jn\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:33:35.726712 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:35.726408 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02d3ea7e-ea37-4970-8afa-2040144cbb89-istio-data\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:33:35.726712 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:35.726418 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02d3ea7e-ea37-4970-8afa-2040144cbb89-istiod-ca-cert\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:33:36.536595 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:36.536558 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-72ce60-56fc9c7fb7-xfqrk" event={"ID":"bcd38542-3ecd-4d77-a6dd-af4503096eca","Type":"ContainerStarted","Data":"8fb8f3435f3b03f03d168bc23cf523b757d33082ee1d0efe791f82c38a743f80"} Jul 19 08:33:36.536846 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:36.536626 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-72ce60-56fc9c7fb7-xfqrk" event={"ID":"bcd38542-3ecd-4d77-a6dd-af4503096eca","Type":"ContainerStarted","Data":"3259f1b10ed003151440d5c03ee7311fbda8eb0e21ab530812fdac98e10c1497"} Jul 19 08:33:36.536846 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:36.536675 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-72ce60-56fc9c7fb7-xfqrk" Jul 19 08:33:36.538345 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:36.538320 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2d2deb-openshift-default-79d58c4cbc-km8cv" Jul 19 08:33:36.553492 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:36.553446 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-72ce60-56fc9c7fb7-xfqrk" podStartSLOduration=2.233685181 podStartE2EDuration="2.553428754s" totalImagesPullingTime="319.743573ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 08:33:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:33:36.552291025 +0000 UTC m=+3082.906600626" watchObservedRunningTime="2026-07-19 08:33:36.553428754 +0000 UTC m=+3082.907738548" Jul 19 08:33:36.581367 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:36.581334 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-2d2deb-openshift-default-79d58c4cbc-km8cv"] Jul 19 08:33:36.584657 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:36.584629 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-2d2deb-openshift-default-79d58c4cbc-km8cv"] Jul 19 08:33:38.561461 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:38.561426 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02d3ea7e-ea37-4970-8afa-2040144cbb89" path="/var/lib/kubelet/pods/02d3ea7e-ea37-4970-8afa-2040144cbb89/volumes" Jul 19 08:33:38.906636 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:38.906511 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:33:39.271905 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:39.271865 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:33:39.474087 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:39.474051 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 08:33:39.554090 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:39.552999 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-190fde-openshift-default-5c6c4d7755-8dlqr"] Jul 19 08:33:39.555932 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:39.555907 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02d3ea7e-ea37-4970-8afa-2040144cbb89" containerName="istio-proxy" Jul 19 08:33:39.556108 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:39.556096 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="02d3ea7e-ea37-4970-8afa-2040144cbb89" containerName="istio-proxy" Jul 19 08:33:39.556446 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:39.556417 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="02d3ea7e-ea37-4970-8afa-2040144cbb89" containerName="istio-proxy" Jul 19 08:33:39.556582 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:39.556571 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="02d3ea7e-ea37-4970-8afa-2040144cbb89" containerName="istio-proxy" Jul 19 08:33:39.561779 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:39.561754 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-190fde-openshift-default-5c6c4d7755-8dlqr" Jul 19 08:33:39.566900 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:39.566871 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-190fde-openshift-default-dockercfg-f2gl8\"" Jul 19 08:33:39.570542 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:39.570367 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:33:39.573105 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:39.573081 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-190fde-openshift-default-5c6c4d7755-8dlqr"] Jul 19 08:33:39.669273 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:39.669231 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81baf455-fff0-45eb-aaf0-ee32450d271b-istiod-ca-cert\") pod \"e2e-shared-b-190fde-openshift-default-5c6c4d7755-8dlqr\" (UID: \"81baf455-fff0-45eb-aaf0-ee32450d271b\") " pod="openshift-ingress/e2e-shared-b-190fde-openshift-default-5c6c4d7755-8dlqr" Jul 19 08:33:39.669509 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:39.669297 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81baf455-fff0-45eb-aaf0-ee32450d271b-istio-data\") pod \"e2e-shared-b-190fde-openshift-default-5c6c4d7755-8dlqr\" (UID: \"81baf455-fff0-45eb-aaf0-ee32450d271b\") " pod="openshift-ingress/e2e-shared-b-190fde-openshift-default-5c6c4d7755-8dlqr" Jul 19 08:33:39.669509 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:39.669369 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81baf455-fff0-45eb-aaf0-ee32450d271b-credential-socket\") pod \"e2e-shared-b-190fde-openshift-default-5c6c4d7755-8dlqr\" (UID: \"81baf455-fff0-45eb-aaf0-ee32450d271b\") " pod="openshift-ingress/e2e-shared-b-190fde-openshift-default-5c6c4d7755-8dlqr" Jul 19 08:33:39.669509 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:39.669414 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81baf455-fff0-45eb-aaf0-ee32450d271b-workload-certs\") pod \"e2e-shared-b-190fde-openshift-default-5c6c4d7755-8dlqr\" (UID: \"81baf455-fff0-45eb-aaf0-ee32450d271b\") " pod="openshift-ingress/e2e-shared-b-190fde-openshift-default-5c6c4d7755-8dlqr" Jul 19 08:33:39.669509 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:39.669489 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81baf455-fff0-45eb-aaf0-ee32450d271b-istio-token\") pod \"e2e-shared-b-190fde-openshift-default-5c6c4d7755-8dlqr\" (UID: \"81baf455-fff0-45eb-aaf0-ee32450d271b\") " pod="openshift-ingress/e2e-shared-b-190fde-openshift-default-5c6c4d7755-8dlqr" Jul 19 08:33:39.669849 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:39.669569 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81baf455-fff0-45eb-aaf0-ee32450d271b-workload-socket\") pod \"e2e-shared-b-190fde-openshift-default-5c6c4d7755-8dlqr\" (UID: \"81baf455-fff0-45eb-aaf0-ee32450d271b\") " pod="openshift-ingress/e2e-shared-b-190fde-openshift-default-5c6c4d7755-8dlqr" Jul 19 08:33:39.669849 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:39.669627 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81baf455-fff0-45eb-aaf0-ee32450d271b-istio-podinfo\") pod \"e2e-shared-b-190fde-openshift-default-5c6c4d7755-8dlqr\" (UID: \"81baf455-fff0-45eb-aaf0-ee32450d271b\") " pod="openshift-ingress/e2e-shared-b-190fde-openshift-default-5c6c4d7755-8dlqr" Jul 19 08:33:39.669849 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:39.669667 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8bpl\" (UniqueName: \"kubernetes.io/projected/81baf455-fff0-45eb-aaf0-ee32450d271b-kube-api-access-k8bpl\") pod \"e2e-shared-b-190fde-openshift-default-5c6c4d7755-8dlqr\" (UID: \"81baf455-fff0-45eb-aaf0-ee32450d271b\") " pod="openshift-ingress/e2e-shared-b-190fde-openshift-default-5c6c4d7755-8dlqr" Jul 19 08:33:39.669849 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:39.669698 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81baf455-fff0-45eb-aaf0-ee32450d271b-istio-envoy\") pod \"e2e-shared-b-190fde-openshift-default-5c6c4d7755-8dlqr\" (UID: \"81baf455-fff0-45eb-aaf0-ee32450d271b\") " pod="openshift-ingress/e2e-shared-b-190fde-openshift-default-5c6c4d7755-8dlqr" Jul 19 08:33:39.770300 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:39.770264 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81baf455-fff0-45eb-aaf0-ee32450d271b-istio-podinfo\") pod \"e2e-shared-b-190fde-openshift-default-5c6c4d7755-8dlqr\" (UID: \"81baf455-fff0-45eb-aaf0-ee32450d271b\") " pod="openshift-ingress/e2e-shared-b-190fde-openshift-default-5c6c4d7755-8dlqr" Jul 19 08:33:39.770300 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:39.770307 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8bpl\" (UniqueName: \"kubernetes.io/projected/81baf455-fff0-45eb-aaf0-ee32450d271b-kube-api-access-k8bpl\") pod \"e2e-shared-b-190fde-openshift-default-5c6c4d7755-8dlqr\" (UID: \"81baf455-fff0-45eb-aaf0-ee32450d271b\") " pod="openshift-ingress/e2e-shared-b-190fde-openshift-default-5c6c4d7755-8dlqr" Jul 19 08:33:39.770611 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:39.770333 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81baf455-fff0-45eb-aaf0-ee32450d271b-istio-envoy\") pod \"e2e-shared-b-190fde-openshift-default-5c6c4d7755-8dlqr\" (UID: \"81baf455-fff0-45eb-aaf0-ee32450d271b\") " pod="openshift-ingress/e2e-shared-b-190fde-openshift-default-5c6c4d7755-8dlqr" Jul 19 08:33:39.770611 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:39.770367 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81baf455-fff0-45eb-aaf0-ee32450d271b-istiod-ca-cert\") pod \"e2e-shared-b-190fde-openshift-default-5c6c4d7755-8dlqr\" (UID: \"81baf455-fff0-45eb-aaf0-ee32450d271b\") " pod="openshift-ingress/e2e-shared-b-190fde-openshift-default-5c6c4d7755-8dlqr" Jul 19 08:33:39.770611 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:39.770395 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81baf455-fff0-45eb-aaf0-ee32450d271b-istio-data\") pod \"e2e-shared-b-190fde-openshift-default-5c6c4d7755-8dlqr\" (UID: \"81baf455-fff0-45eb-aaf0-ee32450d271b\") " pod="openshift-ingress/e2e-shared-b-190fde-openshift-default-5c6c4d7755-8dlqr" Jul 19 08:33:39.770611 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:39.770426 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81baf455-fff0-45eb-aaf0-ee32450d271b-credential-socket\") pod \"e2e-shared-b-190fde-openshift-default-5c6c4d7755-8dlqr\" (UID: \"81baf455-fff0-45eb-aaf0-ee32450d271b\") " pod="openshift-ingress/e2e-shared-b-190fde-openshift-default-5c6c4d7755-8dlqr" Jul 19 08:33:39.770611 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:39.770452 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81baf455-fff0-45eb-aaf0-ee32450d271b-workload-certs\") pod \"e2e-shared-b-190fde-openshift-default-5c6c4d7755-8dlqr\" (UID: \"81baf455-fff0-45eb-aaf0-ee32450d271b\") " pod="openshift-ingress/e2e-shared-b-190fde-openshift-default-5c6c4d7755-8dlqr" Jul 19 08:33:39.770611 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:39.770551 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81baf455-fff0-45eb-aaf0-ee32450d271b-istio-token\") pod \"e2e-shared-b-190fde-openshift-default-5c6c4d7755-8dlqr\" (UID: \"81baf455-fff0-45eb-aaf0-ee32450d271b\") " pod="openshift-ingress/e2e-shared-b-190fde-openshift-default-5c6c4d7755-8dlqr" Jul 19 08:33:39.770611 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:39.770592 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81baf455-fff0-45eb-aaf0-ee32450d271b-workload-socket\") pod \"e2e-shared-b-190fde-openshift-default-5c6c4d7755-8dlqr\" (UID: \"81baf455-fff0-45eb-aaf0-ee32450d271b\") " pod="openshift-ingress/e2e-shared-b-190fde-openshift-default-5c6c4d7755-8dlqr" Jul 19 08:33:39.770983 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:39.770920 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81baf455-fff0-45eb-aaf0-ee32450d271b-credential-socket\") pod \"e2e-shared-b-190fde-openshift-default-5c6c4d7755-8dlqr\" (UID: \"81baf455-fff0-45eb-aaf0-ee32450d271b\") " pod="openshift-ingress/e2e-shared-b-190fde-openshift-default-5c6c4d7755-8dlqr" Jul 19 08:33:39.771126 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:39.771103 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81baf455-fff0-45eb-aaf0-ee32450d271b-istio-data\") pod \"e2e-shared-b-190fde-openshift-default-5c6c4d7755-8dlqr\" (UID: \"81baf455-fff0-45eb-aaf0-ee32450d271b\") " pod="openshift-ingress/e2e-shared-b-190fde-openshift-default-5c6c4d7755-8dlqr" Jul 19 08:33:39.771201 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:39.771139 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81baf455-fff0-45eb-aaf0-ee32450d271b-workload-socket\") pod \"e2e-shared-b-190fde-openshift-default-5c6c4d7755-8dlqr\" (UID: \"81baf455-fff0-45eb-aaf0-ee32450d271b\") " pod="openshift-ingress/e2e-shared-b-190fde-openshift-default-5c6c4d7755-8dlqr" Jul 19 08:33:39.771201 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:39.771180 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81baf455-fff0-45eb-aaf0-ee32450d271b-workload-certs\") pod \"e2e-shared-b-190fde-openshift-default-5c6c4d7755-8dlqr\" (UID: \"81baf455-fff0-45eb-aaf0-ee32450d271b\") " pod="openshift-ingress/e2e-shared-b-190fde-openshift-default-5c6c4d7755-8dlqr" Jul 19 08:33:39.771554 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:39.771514 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81baf455-fff0-45eb-aaf0-ee32450d271b-istiod-ca-cert\") pod \"e2e-shared-b-190fde-openshift-default-5c6c4d7755-8dlqr\" (UID: \"81baf455-fff0-45eb-aaf0-ee32450d271b\") " pod="openshift-ingress/e2e-shared-b-190fde-openshift-default-5c6c4d7755-8dlqr" Jul 19 08:33:39.773226 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:39.773199 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81baf455-fff0-45eb-aaf0-ee32450d271b-istio-envoy\") pod \"e2e-shared-b-190fde-openshift-default-5c6c4d7755-8dlqr\" (UID: \"81baf455-fff0-45eb-aaf0-ee32450d271b\") " pod="openshift-ingress/e2e-shared-b-190fde-openshift-default-5c6c4d7755-8dlqr" Jul 19 08:33:39.773566 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:39.773545 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81baf455-fff0-45eb-aaf0-ee32450d271b-istio-podinfo\") pod \"e2e-shared-b-190fde-openshift-default-5c6c4d7755-8dlqr\" (UID: \"81baf455-fff0-45eb-aaf0-ee32450d271b\") " pod="openshift-ingress/e2e-shared-b-190fde-openshift-default-5c6c4d7755-8dlqr" Jul 19 08:33:39.778123 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:39.778096 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81baf455-fff0-45eb-aaf0-ee32450d271b-istio-token\") pod \"e2e-shared-b-190fde-openshift-default-5c6c4d7755-8dlqr\" (UID: \"81baf455-fff0-45eb-aaf0-ee32450d271b\") " pod="openshift-ingress/e2e-shared-b-190fde-openshift-default-5c6c4d7755-8dlqr" Jul 19 08:33:39.778296 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:39.778275 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8bpl\" (UniqueName: \"kubernetes.io/projected/81baf455-fff0-45eb-aaf0-ee32450d271b-kube-api-access-k8bpl\") pod \"e2e-shared-b-190fde-openshift-default-5c6c4d7755-8dlqr\" (UID: \"81baf455-fff0-45eb-aaf0-ee32450d271b\") " pod="openshift-ingress/e2e-shared-b-190fde-openshift-default-5c6c4d7755-8dlqr" Jul 19 08:33:39.882510 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:39.882416 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-190fde-openshift-default-5c6c4d7755-8dlqr" Jul 19 08:33:39.938680 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:39.938387 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:33:40.033129 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:40.032954 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-190fde-openshift-default-5c6c4d7755-8dlqr"] Jul 19 08:33:40.035566 ip-10-0-136-70 kubenswrapper[2574]: W0719 08:33:40.035507 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81baf455_fff0_45eb_aaf0_ee32450d271b.slice/crio-87dc151194ee7e9f067bb60c5fc67d30e5b13252d31c4b8014110f734a7a37eb WatchSource:0}: Error finding container 87dc151194ee7e9f067bb60c5fc67d30e5b13252d31c4b8014110f734a7a37eb: Status 404 returned error can't find the container with id 87dc151194ee7e9f067bb60c5fc67d30e5b13252d31c4b8014110f734a7a37eb Jul 19 08:33:40.037968 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:40.037918 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 19 08:33:40.038064 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:40.038018 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 19 08:33:40.038106 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:40.038065 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 19 08:33:40.579948 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:40.579917 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-190fde-openshift-default-5c6c4d7755-8dlqr" event={"ID":"81baf455-fff0-45eb-aaf0-ee32450d271b","Type":"ContainerStarted","Data":"769a546ce6f3cd95a34565cd9a64ef428b4d6978960c9b25dfcdf8edc8ba2992"} Jul 19 08:33:40.580383 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:40.579966 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-190fde-openshift-default-5c6c4d7755-8dlqr" event={"ID":"81baf455-fff0-45eb-aaf0-ee32450d271b","Type":"ContainerStarted","Data":"87dc151194ee7e9f067bb60c5fc67d30e5b13252d31c4b8014110f734a7a37eb"} Jul 19 08:33:40.599631 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:40.599582 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-190fde-openshift-default-5c6c4d7755-8dlqr" podStartSLOduration=1.5995637280000001 podStartE2EDuration="1.599563728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 08:33:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:33:40.59741169 +0000 UTC m=+3086.951721305" watchObservedRunningTime="2026-07-19 08:33:40.599563728 +0000 UTC m=+3086.953873307" Jul 19 08:33:40.882897 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:40.882807 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-190fde-openshift-default-5c6c4d7755-8dlqr" Jul 19 08:33:41.888831 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:41.888794 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-190fde-openshift-default-5c6c4d7755-8dlqr" Jul 19 08:33:42.522381 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:42.522344 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:33:42.560113 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:42.560083 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-72ce60-56fc9c7fb7-xfqrk" Jul 19 08:33:42.590907 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:42.590871 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-190fde-openshift-default-5c6c4d7755-8dlqr" Jul 19 08:33:42.591871 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:42.591847 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-190fde-openshift-default-5c6c4d7755-8dlqr" Jul 19 08:33:42.652467 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:42.652436 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:33:42.718173 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:42.718120 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:33:42.924435 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:42.924335 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:33:43.211765 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:43.211731 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:33:43.312296 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:43.312252 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:33:46.987618 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:46.987582 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:33:47.618202 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:47.618167 2574 generic.go:356] "Generic (PLEG): container finished" podID="6c3b1d54-8921-4761-b776-182f6c7889b1" containerID="66a22f07cd26556bb17a657578b5fb7f2ac38eeed2323099571e4ffb271caf7f" exitCode=0 Jul 19 08:33:47.618463 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:47.618243 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-72ce60-openshift-default-7fdc94f55-24snn" event={"ID":"6c3b1d54-8921-4761-b776-182f6c7889b1","Type":"ContainerDied","Data":"66a22f07cd26556bb17a657578b5fb7f2ac38eeed2323099571e4ffb271caf7f"} Jul 19 08:33:47.618691 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:47.618673 2574 scope.go:117] "RemoveContainer" containerID="66a22f07cd26556bb17a657578b5fb7f2ac38eeed2323099571e4ffb271caf7f" Jul 19 08:33:47.619700 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:47.619669 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 19 08:33:47.619764 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:47.619746 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 19 08:33:47.619805 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:47.619797 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 19 08:33:48.514097 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:48.514039 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-72ce60-openshift-default-7fdc94f55-24snn" Jul 19 08:33:48.629390 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:48.629345 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-72ce60-openshift-default-7fdc94f55-24snn" event={"ID":"6c3b1d54-8921-4761-b776-182f6c7889b1","Type":"ContainerStarted","Data":"a3e817b82e5da98b1316e332090ec8116366bffb25437bbfe901af7338e748bd"} Jul 19 08:33:49.430590 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:49.430555 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-72ce60-openshift-default-7fdc94f55-24snn" Jul 19 08:33:49.431921 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:49.431886 2574 patch_prober.go:28] interesting pod/e2e-shared-a-72ce60-openshift-default-7fdc94f55-24snn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Jul 19 08:33:49.432493 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:49.432173 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-72ce60-openshift-default-7fdc94f55-24snn" podUID="6c3b1d54-8921-4761-b776-182f6c7889b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Jul 19 08:33:49.879122 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:49.879078 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 08:33:50.431266 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:50.431222 2574 patch_prober.go:28] interesting pod/e2e-shared-a-72ce60-openshift-default-7fdc94f55-24snn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Jul 19 08:33:50.431615 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:50.431580 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-72ce60-openshift-default-7fdc94f55-24snn" podUID="6c3b1d54-8921-4761-b776-182f6c7889b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Jul 19 08:33:51.430977 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:51.430930 2574 patch_prober.go:28] interesting pod/e2e-shared-a-72ce60-openshift-default-7fdc94f55-24snn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Jul 19 08:33:51.431497 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:51.431004 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-72ce60-openshift-default-7fdc94f55-24snn" podUID="6c3b1d54-8921-4761-b776-182f6c7889b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Jul 19 08:33:52.452829 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:52.452785 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-72ce60-openshift-default-7fdc94f55-24snn" Jul 19 08:33:52.453348 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:52.453326 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-72ce60-openshift-default-7fdc94f55-24snn" Jul 19 08:33:52.455230 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:52.455198 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-72ce60-openshift-default-7fdc94f55-24snn" Jul 19 08:33:52.628689 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:52.628587 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:33:52.718702 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:52.718662 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:33:56.791619 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:56.791580 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:33:56.892579 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:56.892545 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:33:57.973548 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:57.972856 2574 patch_prober.go:28] interesting pod/e2e-shared-b-190fde-openshift-default-5c6c4d7755-8dlqr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": read tcp 10.132.0.2:42904->10.132.0.76:15021: read: connection reset by peer" start-of-body= Jul 19 08:33:57.973548 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:57.972933 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-190fde-openshift-default-5c6c4d7755-8dlqr" podUID="81baf455-fff0-45eb-aaf0-ee32450d271b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": read tcp 10.132.0.2:42904->10.132.0.76:15021: read: connection reset by peer" Jul 19 08:33:58.692183 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:58.692148 2574 generic.go:356] "Generic (PLEG): container finished" podID="81baf455-fff0-45eb-aaf0-ee32450d271b" containerID="769a546ce6f3cd95a34565cd9a64ef428b4d6978960c9b25dfcdf8edc8ba2992" exitCode=0 Jul 19 08:33:58.692365 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:58.692201 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-190fde-openshift-default-5c6c4d7755-8dlqr" event={"ID":"81baf455-fff0-45eb-aaf0-ee32450d271b","Type":"ContainerDied","Data":"769a546ce6f3cd95a34565cd9a64ef428b4d6978960c9b25dfcdf8edc8ba2992"} Jul 19 08:33:58.692647 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:58.692632 2574 scope.go:117] "RemoveContainer" containerID="769a546ce6f3cd95a34565cd9a64ef428b4d6978960c9b25dfcdf8edc8ba2992" Jul 19 08:33:58.693671 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:58.693641 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 19 08:33:58.693733 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:58.693721 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 19 08:33:58.693768 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:58.693754 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 19 08:33:59.699440 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:59.699398 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-190fde-openshift-default-5c6c4d7755-8dlqr" event={"ID":"81baf455-fff0-45eb-aaf0-ee32450d271b","Type":"ContainerStarted","Data":"4d234c9700e4eec23ca8787bc2aa7bb4710593c0bae08422cb41800b89cf68a8"} Jul 19 08:33:59.883353 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:59.883313 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-190fde-openshift-default-5c6c4d7755-8dlqr" Jul 19 08:33:59.883630 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:59.883607 2574 patch_prober.go:28] interesting pod/e2e-shared-b-190fde-openshift-default-5c6c4d7755-8dlqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Jul 19 08:33:59.883737 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:33:59.883663 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-190fde-openshift-default-5c6c4d7755-8dlqr" podUID="81baf455-fff0-45eb-aaf0-ee32450d271b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Jul 19 08:34:00.883379 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:34:00.883337 2574 patch_prober.go:28] interesting pod/e2e-shared-b-190fde-openshift-default-5c6c4d7755-8dlqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Jul 19 08:34:00.883890 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:34:00.883400 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-190fde-openshift-default-5c6c4d7755-8dlqr" podUID="81baf455-fff0-45eb-aaf0-ee32450d271b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Jul 19 08:34:01.883704 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:34:01.883660 2574 patch_prober.go:28] interesting pod/e2e-shared-b-190fde-openshift-default-5c6c4d7755-8dlqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Jul 19 08:34:01.884218 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:34:01.883726 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-190fde-openshift-default-5c6c4d7755-8dlqr" podUID="81baf455-fff0-45eb-aaf0-ee32450d271b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Jul 19 08:34:02.883018 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:34:02.882973 2574 patch_prober.go:28] interesting pod/e2e-shared-b-190fde-openshift-default-5c6c4d7755-8dlqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Jul 19 08:34:02.883208 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:34:02.883053 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-190fde-openshift-default-5c6c4d7755-8dlqr" podUID="81baf455-fff0-45eb-aaf0-ee32450d271b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Jul 19 08:34:03.886778 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:34:03.886747 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-190fde-openshift-default-5c6c4d7755-8dlqr" Jul 19 08:34:03.887208 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:34:03.887125 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-190fde-openshift-default-5c6c4d7755-8dlqr" Jul 19 08:34:03.887774 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:34:03.887754 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-190fde-openshift-default-5c6c4d7755-8dlqr" Jul 19 08:34:03.981625 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:34:03.981590 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:34:05.525739 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:34:05.525696 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:34:06.195663 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:34:06.195624 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:34:09.471957 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:34:09.471913 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 08:34:29.403704 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:34:29.403657 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:34:33.171107 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:34:33.171065 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 08:34:38.133710 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:34:38.133620 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:34:38.188750 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:34:38.188710 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:34:54.645217 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:34:54.645179 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:34:56.375300 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:34:56.375258 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 08:34:58.043646 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:34:58.043606 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-q7bld"] Jul 19 08:34:58.044085 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:34:58.044069 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02d3ea7e-ea37-4970-8afa-2040144cbb89" containerName="istio-proxy" Jul 19 08:34:58.044133 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:34:58.044086 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="02d3ea7e-ea37-4970-8afa-2040144cbb89" containerName="istio-proxy" Jul 19 08:34:58.047378 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:34:58.047354 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-q7bld" Jul 19 08:34:58.054727 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:34:58.054693 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-q7bld"] Jul 19 08:34:58.191866 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:34:58.191824 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t88cf\" (UniqueName: \"kubernetes.io/projected/4c1493cf-0bfd-48c6-a77a-57823e2b4a11-kube-api-access-t88cf\") pod \"kuadrant-operator-catalog-q7bld\" (UID: \"4c1493cf-0bfd-48c6-a77a-57823e2b4a11\") " pod="kuadrant-system/kuadrant-operator-catalog-q7bld" Jul 19 08:34:58.293196 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:34:58.293156 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t88cf\" (UniqueName: \"kubernetes.io/projected/4c1493cf-0bfd-48c6-a77a-57823e2b4a11-kube-api-access-t88cf\") pod \"kuadrant-operator-catalog-q7bld\" (UID: \"4c1493cf-0bfd-48c6-a77a-57823e2b4a11\") " pod="kuadrant-system/kuadrant-operator-catalog-q7bld" Jul 19 08:34:58.301940 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:34:58.301865 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t88cf\" (UniqueName: \"kubernetes.io/projected/4c1493cf-0bfd-48c6-a77a-57823e2b4a11-kube-api-access-t88cf\") pod \"kuadrant-operator-catalog-q7bld\" (UID: \"4c1493cf-0bfd-48c6-a77a-57823e2b4a11\") " pod="kuadrant-system/kuadrant-operator-catalog-q7bld" Jul 19 08:34:58.359974 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:34:58.359940 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-q7bld" Jul 19 08:34:58.495774 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:34:58.495747 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-q7bld"] Jul 19 08:34:58.498637 ip-10-0-136-70 kubenswrapper[2574]: W0719 08:34:58.498604 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c1493cf_0bfd_48c6_a77a_57823e2b4a11.slice/crio-df2166eb57738ed3fb05523e42e29a9286dd990667168b586414b3daf9db5e24 WatchSource:0}: Error finding container df2166eb57738ed3fb05523e42e29a9286dd990667168b586414b3daf9db5e24: Status 404 returned error can't find the container with id df2166eb57738ed3fb05523e42e29a9286dd990667168b586414b3daf9db5e24 Jul 19 08:34:59.024286 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:34:59.024245 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-q7bld" event={"ID":"4c1493cf-0bfd-48c6-a77a-57823e2b4a11","Type":"ContainerStarted","Data":"b451b51202aa2d5b2b01b8b15ad66b3dc30b1eb2261490f8d1cf230723448538"} Jul 19 08:34:59.024508 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:34:59.024303 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-q7bld" event={"ID":"4c1493cf-0bfd-48c6-a77a-57823e2b4a11","Type":"ContainerStarted","Data":"df2166eb57738ed3fb05523e42e29a9286dd990667168b586414b3daf9db5e24"} Jul 19 08:34:59.043883 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:34:59.043820 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-q7bld" podStartSLOduration=0.643328288 podStartE2EDuration="1.043797538s" totalImagesPullingTime="400.46925ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 08:34:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:34:59.039795003 +0000 UTC m=+3165.394104606" watchObservedRunningTime="2026-07-19 08:34:59.043797538 +0000 UTC m=+3165.398107140" Jul 19 08:35:08.360713 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:35:08.360673 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-q7bld" Jul 19 08:35:08.361266 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:35:08.360738 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-q7bld" Jul 19 08:35:08.383728 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:35:08.383700 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-q7bld" Jul 19 08:35:09.105175 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:35:09.105142 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-q7bld" Jul 19 08:35:09.142169 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:35:09.142131 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-q7bld"] Jul 19 08:35:11.092278 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:35:11.092234 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-q7bld" podUID="4c1493cf-0bfd-48c6-a77a-57823e2b4a11" containerName="registry-server" containerID="cri-o://b451b51202aa2d5b2b01b8b15ad66b3dc30b1eb2261490f8d1cf230723448538" gracePeriod=2 Jul 19 08:35:11.351368 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:35:11.351308 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-q7bld" Jul 19 08:35:11.404357 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:35:11.404326 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t88cf\" (UniqueName: \"kubernetes.io/projected/4c1493cf-0bfd-48c6-a77a-57823e2b4a11-kube-api-access-t88cf\") pod \"4c1493cf-0bfd-48c6-a77a-57823e2b4a11\" (UID: \"4c1493cf-0bfd-48c6-a77a-57823e2b4a11\") " Jul 19 08:35:11.406449 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:35:11.406418 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c1493cf-0bfd-48c6-a77a-57823e2b4a11-kube-api-access-t88cf" (OuterVolumeSpecName: "kube-api-access-t88cf") pod "4c1493cf-0bfd-48c6-a77a-57823e2b4a11" (UID: "4c1493cf-0bfd-48c6-a77a-57823e2b4a11"). InnerVolumeSpecName "kube-api-access-t88cf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 08:35:11.505112 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:35:11.505059 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t88cf\" (UniqueName: \"kubernetes.io/projected/4c1493cf-0bfd-48c6-a77a-57823e2b4a11-kube-api-access-t88cf\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:35:12.104483 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:35:12.104448 2574 generic.go:356] "Generic (PLEG): container finished" podID="4c1493cf-0bfd-48c6-a77a-57823e2b4a11" containerID="b451b51202aa2d5b2b01b8b15ad66b3dc30b1eb2261490f8d1cf230723448538" exitCode=0 Jul 19 08:35:12.104945 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:35:12.104495 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-q7bld" event={"ID":"4c1493cf-0bfd-48c6-a77a-57823e2b4a11","Type":"ContainerDied","Data":"b451b51202aa2d5b2b01b8b15ad66b3dc30b1eb2261490f8d1cf230723448538"} Jul 19 08:35:12.104945 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:35:12.104528 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-q7bld" Jul 19 08:35:12.104945 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:35:12.104584 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-q7bld" event={"ID":"4c1493cf-0bfd-48c6-a77a-57823e2b4a11","Type":"ContainerDied","Data":"df2166eb57738ed3fb05523e42e29a9286dd990667168b586414b3daf9db5e24"} Jul 19 08:35:12.104945 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:35:12.104604 2574 scope.go:117] "RemoveContainer" containerID="b451b51202aa2d5b2b01b8b15ad66b3dc30b1eb2261490f8d1cf230723448538" Jul 19 08:35:12.114498 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:35:12.114475 2574 scope.go:117] "RemoveContainer" containerID="b451b51202aa2d5b2b01b8b15ad66b3dc30b1eb2261490f8d1cf230723448538" Jul 19 08:35:12.114816 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:35:12.114796 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b451b51202aa2d5b2b01b8b15ad66b3dc30b1eb2261490f8d1cf230723448538\": container with ID starting with b451b51202aa2d5b2b01b8b15ad66b3dc30b1eb2261490f8d1cf230723448538 not found: ID does not exist" containerID="b451b51202aa2d5b2b01b8b15ad66b3dc30b1eb2261490f8d1cf230723448538" Jul 19 08:35:12.114886 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:35:12.114829 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b451b51202aa2d5b2b01b8b15ad66b3dc30b1eb2261490f8d1cf230723448538"} err="failed to get container status \"b451b51202aa2d5b2b01b8b15ad66b3dc30b1eb2261490f8d1cf230723448538\": rpc error: code = NotFound desc = could not find container \"b451b51202aa2d5b2b01b8b15ad66b3dc30b1eb2261490f8d1cf230723448538\": container with ID starting with b451b51202aa2d5b2b01b8b15ad66b3dc30b1eb2261490f8d1cf230723448538 not found: ID does not exist" Jul 19 08:35:12.127860 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:35:12.127828 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-q7bld"] Jul 19 08:35:12.129875 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:35:12.129848 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-q7bld"] Jul 19 08:35:12.558971 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:35:12.558936 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c1493cf-0bfd-48c6-a77a-57823e2b4a11" path="/var/lib/kubelet/pods/4c1493cf-0bfd-48c6-a77a-57823e2b4a11/volumes" Jul 19 08:35:15.571308 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:35:15.571272 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 08:35:32.772865 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:35:32.772822 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 08:35:48.671601 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:35:48.671559 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 08:36:07.971538 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:36:07.971423 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 08:36:25.674479 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:36:25.674441 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 08:36:40.696502 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:36:40.696459 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:36:41.580068 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:36:41.580030 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:36:45.165730 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:36:45.165691 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 08:36:48.308838 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:36:48.308786 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:37:01.977291 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:37:01.977254 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 08:37:14.801541 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:37:14.801490 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wmhfd_dd68d31d-f763-433a-867b-65b12eabac76/console-operator/1.log" Jul 19 08:37:14.810641 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:37:14.810617 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wmhfd_dd68d31d-f763-433a-867b-65b12eabac76/console-operator/1.log" Jul 19 08:37:19.247604 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:37:19.247566 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:37:23.172627 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:37:23.172588 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 08:37:44.073077 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:37:44.072990 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 08:38:04.175318 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:38:04.175276 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 08:38:09.530493 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:38:09.530455 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:38:10.505904 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:38:10.505866 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:38:17.178191 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:38:17.178127 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:38:23.181890 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:38:23.181846 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 08:38:35.662464 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:38:35.662417 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:38:35.709041 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:38:35.708967 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:38:53.412347 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:38:53.412281 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:38:53.511699 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:38:53.511644 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:38:57.531942 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:38:57.531905 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:38:57.632430 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:38:57.632383 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:39:03.404203 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:39:03.404160 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:39:04.807252 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:39:04.807215 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:39:04.908665 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:39:04.908627 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:39:08.373530 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:39:08.373434 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 08:39:15.021736 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:39:15.021691 2574 scope.go:117] "RemoveContainer" containerID="ea134ab394238f6bf2656df69201dd296b5549b8f5550cb1f3110f9d82db34ae" Jul 19 08:39:29.980797 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:39:29.980754 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 08:39:45.872310 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:39:45.872267 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 08:39:52.079402 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:39:52.079354 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:39:53.440244 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:39:53.440207 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:39:53.880979 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:39:53.880943 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7491-140088652168624"] Jul 19 08:39:53.881512 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:39:53.881490 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c1493cf-0bfd-48c6-a77a-57823e2b4a11" containerName="registry-server" Jul 19 08:39:53.881664 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:39:53.881513 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c1493cf-0bfd-48c6-a77a-57823e2b4a11" containerName="registry-server" Jul 19 08:39:53.881664 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:39:53.881658 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c1493cf-0bfd-48c6-a77a-57823e2b4a11" containerName="registry-server" Jul 19 08:39:53.885093 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:39:53.885072 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7491-140088652168624" Jul 19 08:39:53.887812 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:39:53.887793 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-v77hc\"" Jul 19 08:39:53.891176 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:39:53.891149 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7491-140088652168624"] Jul 19 08:39:54.009284 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:39:54.009246 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdkdn\" (UniqueName: \"kubernetes.io/projected/f72a6828-a551-4f5b-932c-0aaa709cd7b4-kube-api-access-pdkdn\") pod \"test-curl-7491-140088652168624\" (UID: \"f72a6828-a551-4f5b-932c-0aaa709cd7b4\") " pod="opendatahub/test-curl-7491-140088652168624" Jul 19 08:39:54.110116 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:39:54.110075 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdkdn\" (UniqueName: \"kubernetes.io/projected/f72a6828-a551-4f5b-932c-0aaa709cd7b4-kube-api-access-pdkdn\") pod \"test-curl-7491-140088652168624\" (UID: \"f72a6828-a551-4f5b-932c-0aaa709cd7b4\") " pod="opendatahub/test-curl-7491-140088652168624" Jul 19 08:39:54.118624 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:39:54.118587 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdkdn\" (UniqueName: \"kubernetes.io/projected/f72a6828-a551-4f5b-932c-0aaa709cd7b4-kube-api-access-pdkdn\") pod \"test-curl-7491-140088652168624\" (UID: \"f72a6828-a551-4f5b-932c-0aaa709cd7b4\") " pod="opendatahub/test-curl-7491-140088652168624" Jul 19 08:39:54.197048 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:39:54.196937 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7491-140088652168624" Jul 19 08:39:54.330955 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:39:54.330927 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7491-140088652168624"] Jul 19 08:39:54.333145 ip-10-0-136-70 kubenswrapper[2574]: W0719 08:39:54.333117 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf72a6828_a551_4f5b_932c_0aaa709cd7b4.slice/crio-72e9d432202e0030e78ae41513f04ee133c61cdd7b2cf3717a9363b97af92218 WatchSource:0}: Error finding container 72e9d432202e0030e78ae41513f04ee133c61cdd7b2cf3717a9363b97af92218: Status 404 returned error can't find the container with id 72e9d432202e0030e78ae41513f04ee133c61cdd7b2cf3717a9363b97af92218 Jul 19 08:39:54.334976 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:39:54.334959 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 19 08:39:54.588802 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:39:54.588764 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7491-140088652168624" event={"ID":"f72a6828-a551-4f5b-932c-0aaa709cd7b4","Type":"ContainerStarted","Data":"72e9d432202e0030e78ae41513f04ee133c61cdd7b2cf3717a9363b97af92218"} Jul 19 08:39:55.595073 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:39:55.595034 2574 generic.go:356] "Generic (PLEG): container finished" podID="f72a6828-a551-4f5b-932c-0aaa709cd7b4" containerID="f97598d608f2bdf4e42c597f7b12d59e944cbbf3b929b17196944219b6ec1e42" exitCode=0 Jul 19 08:39:55.595504 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:39:55.595105 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7491-140088652168624" event={"ID":"f72a6828-a551-4f5b-932c-0aaa709cd7b4","Type":"ContainerDied","Data":"f97598d608f2bdf4e42c597f7b12d59e944cbbf3b929b17196944219b6ec1e42"} Jul 19 08:39:56.742875 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:39:56.742849 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7491-140088652168624" Jul 19 08:39:56.760842 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:39:56.760813 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7491-140088652168624_f72a6828-a551-4f5b-932c-0aaa709cd7b4/test-curl-7491-140088652168624/0.log" Jul 19 08:39:56.781793 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:39:56.781760 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7491-140088652168624"] Jul 19 08:39:56.785709 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:39:56.785679 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7491-140088652168624"] Jul 19 08:39:56.835980 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:39:56.835940 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pdkdn\" (UniqueName: \"kubernetes.io/projected/f72a6828-a551-4f5b-932c-0aaa709cd7b4-kube-api-access-pdkdn\") pod \"f72a6828-a551-4f5b-932c-0aaa709cd7b4\" (UID: \"f72a6828-a551-4f5b-932c-0aaa709cd7b4\") " Jul 19 08:39:56.838264 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:39:56.838234 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f72a6828-a551-4f5b-932c-0aaa709cd7b4-kube-api-access-pdkdn" (OuterVolumeSpecName: "kube-api-access-pdkdn") pod "f72a6828-a551-4f5b-932c-0aaa709cd7b4" (UID: "f72a6828-a551-4f5b-932c-0aaa709cd7b4"). InnerVolumeSpecName "kube-api-access-pdkdn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 08:39:56.903448 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:39:56.903368 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7491-140088652202288"] Jul 19 08:39:56.903848 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:39:56.903833 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f72a6828-a551-4f5b-932c-0aaa709cd7b4" containerName="test-curl-7491-140088652168624" Jul 19 08:39:56.903907 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:39:56.903851 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="f72a6828-a551-4f5b-932c-0aaa709cd7b4" containerName="test-curl-7491-140088652168624" Jul 19 08:39:56.903944 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:39:56.903909 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="f72a6828-a551-4f5b-932c-0aaa709cd7b4" containerName="test-curl-7491-140088652168624" Jul 19 08:39:56.907049 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:39:56.907031 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7491-140088652202288" Jul 19 08:39:56.913548 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:39:56.913506 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7491-140088652202288"] Jul 19 08:39:56.937367 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:39:56.937324 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdkdn\" (UniqueName: \"kubernetes.io/projected/f72a6828-a551-4f5b-932c-0aaa709cd7b4-kube-api-access-pdkdn\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:39:57.038099 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:39:57.038053 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rsq7\" (UniqueName: \"kubernetes.io/projected/27fa97b8-e115-4fc4-8562-d2390f72d7e7-kube-api-access-4rsq7\") pod \"test-curl-7491-140088652202288\" (UID: \"27fa97b8-e115-4fc4-8562-d2390f72d7e7\") " pod="opendatahub/test-curl-7491-140088652202288" Jul 19 08:39:57.138759 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:39:57.138713 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rsq7\" (UniqueName: \"kubernetes.io/projected/27fa97b8-e115-4fc4-8562-d2390f72d7e7-kube-api-access-4rsq7\") pod \"test-curl-7491-140088652202288\" (UID: \"27fa97b8-e115-4fc4-8562-d2390f72d7e7\") " pod="opendatahub/test-curl-7491-140088652202288" Jul 19 08:39:57.147200 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:39:57.147164 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rsq7\" (UniqueName: \"kubernetes.io/projected/27fa97b8-e115-4fc4-8562-d2390f72d7e7-kube-api-access-4rsq7\") pod \"test-curl-7491-140088652202288\" (UID: \"27fa97b8-e115-4fc4-8562-d2390f72d7e7\") " pod="opendatahub/test-curl-7491-140088652202288" Jul 19 08:39:57.219083 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:39:57.219048 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7491-140088652202288" Jul 19 08:39:57.360098 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:39:57.360072 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7491-140088652202288"] Jul 19 08:39:57.362648 ip-10-0-136-70 kubenswrapper[2574]: W0719 08:39:57.362619 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod27fa97b8_e115_4fc4_8562_d2390f72d7e7.slice/crio-f6ebdd0262ab49c8c0c6624a03d5e49f79906caf6bd5aa50a7bfe5f74687e00b WatchSource:0}: Error finding container f6ebdd0262ab49c8c0c6624a03d5e49f79906caf6bd5aa50a7bfe5f74687e00b: Status 404 returned error can't find the container with id f6ebdd0262ab49c8c0c6624a03d5e49f79906caf6bd5aa50a7bfe5f74687e00b Jul 19 08:39:57.606364 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:39:57.606252 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7491-140088652202288" event={"ID":"27fa97b8-e115-4fc4-8562-d2390f72d7e7","Type":"ContainerStarted","Data":"f6ebdd0262ab49c8c0c6624a03d5e49f79906caf6bd5aa50a7bfe5f74687e00b"} Jul 19 08:39:57.607582 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:39:57.607560 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="72e9d432202e0030e78ae41513f04ee133c61cdd7b2cf3717a9363b97af92218" Jul 19 08:39:57.607582 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:39:57.607577 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7491-140088652168624" Jul 19 08:39:57.618614 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:39:57.618582 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7491-140088652168624\" is forbidden: User \"system:node:ip-10-0-136-70.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-70.ec2.internal' and this object" podUID="f72a6828-a551-4f5b-932c-0aaa709cd7b4" pod="opendatahub/test-curl-7491-140088652168624" Jul 19 08:39:58.558625 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:39:58.558587 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f72a6828-a551-4f5b-932c-0aaa709cd7b4" path="/var/lib/kubelet/pods/f72a6828-a551-4f5b-932c-0aaa709cd7b4/volumes" Jul 19 08:39:58.613862 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:39:58.613829 2574 generic.go:356] "Generic (PLEG): container finished" podID="27fa97b8-e115-4fc4-8562-d2390f72d7e7" containerID="22429c356934a71291029e0f95486b9917c2f0326edeceb64738b7e5477a2389" exitCode=0 Jul 19 08:39:58.614019 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:39:58.613897 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7491-140088652202288" event={"ID":"27fa97b8-e115-4fc4-8562-d2390f72d7e7","Type":"ContainerDied","Data":"22429c356934a71291029e0f95486b9917c2f0326edeceb64738b7e5477a2389"} Jul 19 08:39:59.775423 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:39:59.775391 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7491-140088652202288" Jul 19 08:39:59.794365 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:39:59.794333 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7491-140088652202288_27fa97b8-e115-4fc4-8562-d2390f72d7e7/test-curl-7491-140088652202288/0.log" Jul 19 08:39:59.817417 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:39:59.817381 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7491-140088652202288"] Jul 19 08:39:59.819464 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:39:59.819439 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7491-140088652202288"] Jul 19 08:39:59.864177 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:39:59.864142 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4rsq7\" (UniqueName: \"kubernetes.io/projected/27fa97b8-e115-4fc4-8562-d2390f72d7e7-kube-api-access-4rsq7\") pod \"27fa97b8-e115-4fc4-8562-d2390f72d7e7\" (UID: \"27fa97b8-e115-4fc4-8562-d2390f72d7e7\") " Jul 19 08:39:59.866329 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:39:59.866295 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27fa97b8-e115-4fc4-8562-d2390f72d7e7-kube-api-access-4rsq7" (OuterVolumeSpecName: "kube-api-access-4rsq7") pod "27fa97b8-e115-4fc4-8562-d2390f72d7e7" (UID: "27fa97b8-e115-4fc4-8562-d2390f72d7e7"). InnerVolumeSpecName "kube-api-access-4rsq7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 08:39:59.965142 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:39:59.965102 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rsq7\" (UniqueName: \"kubernetes.io/projected/27fa97b8-e115-4fc4-8562-d2390f72d7e7-kube-api-access-4rsq7\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:39:59.971967 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:39:59.971931 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7491-140088652065072"] Jul 19 08:39:59.972424 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:39:59.972408 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27fa97b8-e115-4fc4-8562-d2390f72d7e7" containerName="test-curl-7491-140088652202288" Jul 19 08:39:59.972469 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:39:59.972426 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="27fa97b8-e115-4fc4-8562-d2390f72d7e7" containerName="test-curl-7491-140088652202288" Jul 19 08:39:59.972513 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:39:59.972504 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="27fa97b8-e115-4fc4-8562-d2390f72d7e7" containerName="test-curl-7491-140088652202288" Jul 19 08:39:59.975609 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:39:59.975589 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7491-140088652065072" Jul 19 08:39:59.981684 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:39:59.981652 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7491-140088652065072"] Jul 19 08:40:00.066460 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:00.066413 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvp99\" (UniqueName: \"kubernetes.io/projected/5ac80e76-3679-4512-9162-041b80ffd75b-kube-api-access-rvp99\") pod \"test-curl-7491-140088652065072\" (UID: \"5ac80e76-3679-4512-9162-041b80ffd75b\") " pod="opendatahub/test-curl-7491-140088652065072" Jul 19 08:40:00.168070 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:00.167954 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvp99\" (UniqueName: \"kubernetes.io/projected/5ac80e76-3679-4512-9162-041b80ffd75b-kube-api-access-rvp99\") pod \"test-curl-7491-140088652065072\" (UID: \"5ac80e76-3679-4512-9162-041b80ffd75b\") " pod="opendatahub/test-curl-7491-140088652065072" Jul 19 08:40:00.176051 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:00.176017 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvp99\" (UniqueName: \"kubernetes.io/projected/5ac80e76-3679-4512-9162-041b80ffd75b-kube-api-access-rvp99\") pod \"test-curl-7491-140088652065072\" (UID: \"5ac80e76-3679-4512-9162-041b80ffd75b\") " pod="opendatahub/test-curl-7491-140088652065072" Jul 19 08:40:00.285880 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:00.285836 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7491-140088652065072" Jul 19 08:40:00.422811 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:00.422787 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7491-140088652065072"] Jul 19 08:40:00.426152 ip-10-0-136-70 kubenswrapper[2574]: W0719 08:40:00.426122 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5ac80e76_3679_4512_9162_041b80ffd75b.slice/crio-7367cc7f426cdda6306cf313e4dd281463e69108dfe7d3f8425e247b5dfb884c WatchSource:0}: Error finding container 7367cc7f426cdda6306cf313e4dd281463e69108dfe7d3f8425e247b5dfb884c: Status 404 returned error can't find the container with id 7367cc7f426cdda6306cf313e4dd281463e69108dfe7d3f8425e247b5dfb884c Jul 19 08:40:00.559006 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:00.558962 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27fa97b8-e115-4fc4-8562-d2390f72d7e7" path="/var/lib/kubelet/pods/27fa97b8-e115-4fc4-8562-d2390f72d7e7/volumes" Jul 19 08:40:00.627155 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:00.627119 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7491-140088652202288" Jul 19 08:40:00.627360 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:00.627168 2574 scope.go:117] "RemoveContainer" containerID="22429c356934a71291029e0f95486b9917c2f0326edeceb64738b7e5477a2389" Jul 19 08:40:00.628721 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:00.628557 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7491-140088652065072" event={"ID":"5ac80e76-3679-4512-9162-041b80ffd75b","Type":"ContainerStarted","Data":"7367cc7f426cdda6306cf313e4dd281463e69108dfe7d3f8425e247b5dfb884c"} Jul 19 08:40:01.070446 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:01.070404 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 08:40:01.636894 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:01.636851 2574 generic.go:356] "Generic (PLEG): container finished" podID="5ac80e76-3679-4512-9162-041b80ffd75b" containerID="d17682a5eb291dc0e1accfcf825c780a511c54fa0ceb342583cb50ec1707706a" exitCode=0 Jul 19 08:40:01.636894 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:01.636888 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7491-140088652065072" event={"ID":"5ac80e76-3679-4512-9162-041b80ffd75b","Type":"ContainerDied","Data":"d17682a5eb291dc0e1accfcf825c780a511c54fa0ceb342583cb50ec1707706a"} Jul 19 08:40:02.783052 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:02.783025 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7491-140088652065072" Jul 19 08:40:02.801429 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:02.801393 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7491-140088652065072_5ac80e76-3679-4512-9162-041b80ffd75b/test-curl-7491-140088652065072/0.log" Jul 19 08:40:02.837005 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:02.836965 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7491-140088652065072"] Jul 19 08:40:02.840367 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:02.840338 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7491-140088652065072"] Jul 19 08:40:02.892060 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:02.891959 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rvp99\" (UniqueName: \"kubernetes.io/projected/5ac80e76-3679-4512-9162-041b80ffd75b-kube-api-access-rvp99\") pod \"5ac80e76-3679-4512-9162-041b80ffd75b\" (UID: \"5ac80e76-3679-4512-9162-041b80ffd75b\") " Jul 19 08:40:02.894073 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:02.894051 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ac80e76-3679-4512-9162-041b80ffd75b-kube-api-access-rvp99" (OuterVolumeSpecName: "kube-api-access-rvp99") pod "5ac80e76-3679-4512-9162-041b80ffd75b" (UID: "5ac80e76-3679-4512-9162-041b80ffd75b"). InnerVolumeSpecName "kube-api-access-rvp99". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 08:40:02.953385 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:02.953327 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7491-140088652065072"] Jul 19 08:40:02.953891 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:02.953869 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ac80e76-3679-4512-9162-041b80ffd75b" containerName="test-curl-7491-140088652065072" Jul 19 08:40:02.953891 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:02.953892 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ac80e76-3679-4512-9162-041b80ffd75b" containerName="test-curl-7491-140088652065072" Jul 19 08:40:02.954143 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:02.954034 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ac80e76-3679-4512-9162-041b80ffd75b" containerName="test-curl-7491-140088652065072" Jul 19 08:40:02.957271 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:02.957251 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7491-140088652065072" Jul 19 08:40:02.961608 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:02.961575 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7491-140088652065072"] Jul 19 08:40:02.992809 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:02.992780 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvp99\" (UniqueName: \"kubernetes.io/projected/5ac80e76-3679-4512-9162-041b80ffd75b-kube-api-access-rvp99\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:40:03.093301 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:03.093267 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6tl7\" (UniqueName: \"kubernetes.io/projected/e58316de-34d0-4aaa-a833-1a9ea2dff825-kube-api-access-x6tl7\") pod \"test-curl-7491-140088652065072\" (UID: \"e58316de-34d0-4aaa-a833-1a9ea2dff825\") " pod="opendatahub/test-curl-7491-140088652065072" Jul 19 08:40:03.194663 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:03.194559 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6tl7\" (UniqueName: \"kubernetes.io/projected/e58316de-34d0-4aaa-a833-1a9ea2dff825-kube-api-access-x6tl7\") pod \"test-curl-7491-140088652065072\" (UID: \"e58316de-34d0-4aaa-a833-1a9ea2dff825\") " pod="opendatahub/test-curl-7491-140088652065072" Jul 19 08:40:03.202605 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:03.202577 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6tl7\" (UniqueName: \"kubernetes.io/projected/e58316de-34d0-4aaa-a833-1a9ea2dff825-kube-api-access-x6tl7\") pod \"test-curl-7491-140088652065072\" (UID: \"e58316de-34d0-4aaa-a833-1a9ea2dff825\") " pod="opendatahub/test-curl-7491-140088652065072" Jul 19 08:40:03.268589 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:03.268511 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7491-140088652065072" Jul 19 08:40:03.408352 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:03.408327 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7491-140088652065072"] Jul 19 08:40:03.410508 ip-10-0-136-70 kubenswrapper[2574]: W0719 08:40:03.410474 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode58316de_34d0_4aaa_a833_1a9ea2dff825.slice/crio-20f178f8299147b8ddc159976135410eb29160ab85829f9d8ed03ce33f4d59ea WatchSource:0}: Error finding container 20f178f8299147b8ddc159976135410eb29160ab85829f9d8ed03ce33f4d59ea: Status 404 returned error can't find the container with id 20f178f8299147b8ddc159976135410eb29160ab85829f9d8ed03ce33f4d59ea Jul 19 08:40:03.648201 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:03.648170 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7491-140088652065072" Jul 19 08:40:03.648402 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:03.648217 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7367cc7f426cdda6306cf313e4dd281463e69108dfe7d3f8425e247b5dfb884c" Jul 19 08:40:03.649496 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:03.649466 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7491-140088652065072" event={"ID":"e58316de-34d0-4aaa-a833-1a9ea2dff825","Type":"ContainerStarted","Data":"20f178f8299147b8ddc159976135410eb29160ab85829f9d8ed03ce33f4d59ea"} Jul 19 08:40:03.660261 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:03.660227 2574 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7491-140088652065072" oldPodUID="5ac80e76-3679-4512-9162-041b80ffd75b" podUID="e58316de-34d0-4aaa-a833-1a9ea2dff825" Jul 19 08:40:04.559154 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:04.559111 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ac80e76-3679-4512-9162-041b80ffd75b" path="/var/lib/kubelet/pods/5ac80e76-3679-4512-9162-041b80ffd75b/volumes" Jul 19 08:40:04.656725 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:04.656692 2574 generic.go:356] "Generic (PLEG): container finished" podID="e58316de-34d0-4aaa-a833-1a9ea2dff825" containerID="13cf7f22472405479fb80581d7a61942081bfc4032065cf1a77aa17867c1eacc" exitCode=0 Jul 19 08:40:04.656907 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:04.656749 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7491-140088652065072" event={"ID":"e58316de-34d0-4aaa-a833-1a9ea2dff825","Type":"ContainerDied","Data":"13cf7f22472405479fb80581d7a61942081bfc4032065cf1a77aa17867c1eacc"} Jul 19 08:40:05.807733 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:05.807707 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7491-140088652065072" Jul 19 08:40:05.825812 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:05.825788 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7491-140088652065072_e58316de-34d0-4aaa-a833-1a9ea2dff825/test-curl-7491-140088652065072/0.log" Jul 19 08:40:05.847108 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:05.847072 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7491-140088652065072"] Jul 19 08:40:05.852622 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:05.852588 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7491-140088652065072"] Jul 19 08:40:05.921222 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:05.921113 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x6tl7\" (UniqueName: \"kubernetes.io/projected/e58316de-34d0-4aaa-a833-1a9ea2dff825-kube-api-access-x6tl7\") pod \"e58316de-34d0-4aaa-a833-1a9ea2dff825\" (UID: \"e58316de-34d0-4aaa-a833-1a9ea2dff825\") " Jul 19 08:40:05.923265 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:05.923230 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e58316de-34d0-4aaa-a833-1a9ea2dff825-kube-api-access-x6tl7" (OuterVolumeSpecName: "kube-api-access-x6tl7") pod "e58316de-34d0-4aaa-a833-1a9ea2dff825" (UID: "e58316de-34d0-4aaa-a833-1a9ea2dff825"). InnerVolumeSpecName "kube-api-access-x6tl7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 08:40:05.967202 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:05.967168 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7491-140088652066080"] Jul 19 08:40:05.967649 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:05.967634 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e58316de-34d0-4aaa-a833-1a9ea2dff825" containerName="test-curl-7491-140088652065072" Jul 19 08:40:05.967712 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:05.967650 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e58316de-34d0-4aaa-a833-1a9ea2dff825" containerName="test-curl-7491-140088652065072" Jul 19 08:40:05.967749 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:05.967723 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="e58316de-34d0-4aaa-a833-1a9ea2dff825" containerName="test-curl-7491-140088652065072" Jul 19 08:40:05.970880 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:05.970865 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7491-140088652066080" Jul 19 08:40:05.977504 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:05.977476 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7491-140088652066080"] Jul 19 08:40:06.022010 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:06.021974 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6tl7\" (UniqueName: \"kubernetes.io/projected/e58316de-34d0-4aaa-a833-1a9ea2dff825-kube-api-access-x6tl7\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:40:06.123177 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:06.123132 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npsvz\" (UniqueName: \"kubernetes.io/projected/983b4de7-c21c-4157-b196-8df58dd8eed1-kube-api-access-npsvz\") pod \"test-curl-7491-140088652066080\" (UID: \"983b4de7-c21c-4157-b196-8df58dd8eed1\") " pod="opendatahub/test-curl-7491-140088652066080" Jul 19 08:40:06.224247 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:06.224200 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npsvz\" (UniqueName: \"kubernetes.io/projected/983b4de7-c21c-4157-b196-8df58dd8eed1-kube-api-access-npsvz\") pod \"test-curl-7491-140088652066080\" (UID: \"983b4de7-c21c-4157-b196-8df58dd8eed1\") " pod="opendatahub/test-curl-7491-140088652066080" Jul 19 08:40:06.232202 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:06.232165 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-npsvz\" (UniqueName: \"kubernetes.io/projected/983b4de7-c21c-4157-b196-8df58dd8eed1-kube-api-access-npsvz\") pod \"test-curl-7491-140088652066080\" (UID: \"983b4de7-c21c-4157-b196-8df58dd8eed1\") " pod="opendatahub/test-curl-7491-140088652066080" Jul 19 08:40:06.281645 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:06.281611 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7491-140088652066080" Jul 19 08:40:06.418556 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:06.418514 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7491-140088652066080"] Jul 19 08:40:06.420882 ip-10-0-136-70 kubenswrapper[2574]: W0719 08:40:06.420838 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod983b4de7_c21c_4157_b196_8df58dd8eed1.slice/crio-533cb9a70bba03f9dbf522e70788eb54c7da11921bda1768a5853172fd9c2f51 WatchSource:0}: Error finding container 533cb9a70bba03f9dbf522e70788eb54c7da11921bda1768a5853172fd9c2f51: Status 404 returned error can't find the container with id 533cb9a70bba03f9dbf522e70788eb54c7da11921bda1768a5853172fd9c2f51 Jul 19 08:40:06.562751 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:06.562670 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e58316de-34d0-4aaa-a833-1a9ea2dff825" path="/var/lib/kubelet/pods/e58316de-34d0-4aaa-a833-1a9ea2dff825/volumes" Jul 19 08:40:06.669393 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:06.669356 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7491-140088652066080" event={"ID":"983b4de7-c21c-4157-b196-8df58dd8eed1","Type":"ContainerStarted","Data":"533cb9a70bba03f9dbf522e70788eb54c7da11921bda1768a5853172fd9c2f51"} Jul 19 08:40:06.670740 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:06.670716 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7491-140088652065072" Jul 19 08:40:06.670740 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:06.670715 2574 scope.go:117] "RemoveContainer" containerID="13cf7f22472405479fb80581d7a61942081bfc4032065cf1a77aa17867c1eacc" Jul 19 08:40:07.678223 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:07.678179 2574 generic.go:356] "Generic (PLEG): container finished" podID="983b4de7-c21c-4157-b196-8df58dd8eed1" containerID="c48dfa4621d1328fcba3cc05ed3c3f751be768d2a9947a1a50e4e3905c266f41" exitCode=0 Jul 19 08:40:07.678739 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:07.678273 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7491-140088652066080" event={"ID":"983b4de7-c21c-4157-b196-8df58dd8eed1","Type":"ContainerDied","Data":"c48dfa4621d1328fcba3cc05ed3c3f751be768d2a9947a1a50e4e3905c266f41"} Jul 19 08:40:08.831001 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:08.830973 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7491-140088652066080" Jul 19 08:40:08.851780 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:08.851750 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7491-140088652066080_983b4de7-c21c-4157-b196-8df58dd8eed1/test-curl-7491-140088652066080/0.log" Jul 19 08:40:08.874913 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:08.874779 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7491-140088652066080"] Jul 19 08:40:08.876643 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:08.876624 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7491-140088652066080"] Jul 19 08:40:08.945334 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:08.945221 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-npsvz\" (UniqueName: \"kubernetes.io/projected/983b4de7-c21c-4157-b196-8df58dd8eed1-kube-api-access-npsvz\") pod \"983b4de7-c21c-4157-b196-8df58dd8eed1\" (UID: \"983b4de7-c21c-4157-b196-8df58dd8eed1\") " Jul 19 08:40:08.947355 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:08.947322 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/983b4de7-c21c-4157-b196-8df58dd8eed1-kube-api-access-npsvz" (OuterVolumeSpecName: "kube-api-access-npsvz") pod "983b4de7-c21c-4157-b196-8df58dd8eed1" (UID: "983b4de7-c21c-4157-b196-8df58dd8eed1"). InnerVolumeSpecName "kube-api-access-npsvz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 08:40:08.995994 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:08.995947 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7491-140088652066080"] Jul 19 08:40:08.996463 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:08.996435 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="983b4de7-c21c-4157-b196-8df58dd8eed1" containerName="test-curl-7491-140088652066080" Jul 19 08:40:08.996463 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:08.996456 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="983b4de7-c21c-4157-b196-8df58dd8eed1" containerName="test-curl-7491-140088652066080" Jul 19 08:40:08.996659 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:08.996571 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="983b4de7-c21c-4157-b196-8df58dd8eed1" containerName="test-curl-7491-140088652066080" Jul 19 08:40:08.999784 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:08.999761 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7491-140088652066080" Jul 19 08:40:09.005548 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:09.005500 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7491-140088652066080"] Jul 19 08:40:09.045835 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:09.045801 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkqt5\" (UniqueName: \"kubernetes.io/projected/bb31a31f-73ae-47f6-b8cc-f3429392348f-kube-api-access-dkqt5\") pod \"test-curl-7491-140088652066080\" (UID: \"bb31a31f-73ae-47f6-b8cc-f3429392348f\") " pod="opendatahub/test-curl-7491-140088652066080" Jul 19 08:40:09.045992 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:09.045870 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npsvz\" (UniqueName: \"kubernetes.io/projected/983b4de7-c21c-4157-b196-8df58dd8eed1-kube-api-access-npsvz\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:40:09.146234 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:09.146202 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkqt5\" (UniqueName: \"kubernetes.io/projected/bb31a31f-73ae-47f6-b8cc-f3429392348f-kube-api-access-dkqt5\") pod \"test-curl-7491-140088652066080\" (UID: \"bb31a31f-73ae-47f6-b8cc-f3429392348f\") " pod="opendatahub/test-curl-7491-140088652066080" Jul 19 08:40:09.154076 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:09.154044 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkqt5\" (UniqueName: \"kubernetes.io/projected/bb31a31f-73ae-47f6-b8cc-f3429392348f-kube-api-access-dkqt5\") pod \"test-curl-7491-140088652066080\" (UID: \"bb31a31f-73ae-47f6-b8cc-f3429392348f\") " pod="opendatahub/test-curl-7491-140088652066080" Jul 19 08:40:09.311057 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:09.311019 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7491-140088652066080" Jul 19 08:40:09.446870 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:09.446841 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7491-140088652066080"] Jul 19 08:40:09.448619 ip-10-0-136-70 kubenswrapper[2574]: W0719 08:40:09.448595 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbb31a31f_73ae_47f6_b8cc_f3429392348f.slice/crio-76169deeb7c0d289fc22ebc2d192f97206215df414309b299922e98f02173130 WatchSource:0}: Error finding container 76169deeb7c0d289fc22ebc2d192f97206215df414309b299922e98f02173130: Status 404 returned error can't find the container with id 76169deeb7c0d289fc22ebc2d192f97206215df414309b299922e98f02173130 Jul 19 08:40:09.691609 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:09.691528 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7491-140088652066080" event={"ID":"bb31a31f-73ae-47f6-b8cc-f3429392348f","Type":"ContainerStarted","Data":"76169deeb7c0d289fc22ebc2d192f97206215df414309b299922e98f02173130"} Jul 19 08:40:09.692963 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:09.692937 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7491-140088652066080" Jul 19 08:40:09.693092 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:09.692978 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="533cb9a70bba03f9dbf522e70788eb54c7da11921bda1768a5853172fd9c2f51" Jul 19 08:40:09.704368 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:09.704341 2574 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7491-140088652066080" oldPodUID="983b4de7-c21c-4157-b196-8df58dd8eed1" podUID="bb31a31f-73ae-47f6-b8cc-f3429392348f" Jul 19 08:40:10.563888 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:10.563850 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="983b4de7-c21c-4157-b196-8df58dd8eed1" path="/var/lib/kubelet/pods/983b4de7-c21c-4157-b196-8df58dd8eed1/volumes" Jul 19 08:40:10.701584 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:10.701547 2574 generic.go:356] "Generic (PLEG): container finished" podID="bb31a31f-73ae-47f6-b8cc-f3429392348f" containerID="1b40b9170c259d27080c0381b6545035da5df1cdd16dbc6b07dab948f4ded56d" exitCode=0 Jul 19 08:40:10.701791 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:10.701606 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7491-140088652066080" event={"ID":"bb31a31f-73ae-47f6-b8cc-f3429392348f","Type":"ContainerDied","Data":"1b40b9170c259d27080c0381b6545035da5df1cdd16dbc6b07dab948f4ded56d"} Jul 19 08:40:11.852662 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:11.852636 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7491-140088652066080" Jul 19 08:40:11.868026 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:11.867994 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dkqt5\" (UniqueName: \"kubernetes.io/projected/bb31a31f-73ae-47f6-b8cc-f3429392348f-kube-api-access-dkqt5\") pod \"bb31a31f-73ae-47f6-b8cc-f3429392348f\" (UID: \"bb31a31f-73ae-47f6-b8cc-f3429392348f\") " Jul 19 08:40:11.870694 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:11.870654 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb31a31f-73ae-47f6-b8cc-f3429392348f-kube-api-access-dkqt5" (OuterVolumeSpecName: "kube-api-access-dkqt5") pod "bb31a31f-73ae-47f6-b8cc-f3429392348f" (UID: "bb31a31f-73ae-47f6-b8cc-f3429392348f"). InnerVolumeSpecName "kube-api-access-dkqt5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 08:40:11.871162 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:11.871129 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7491-140088652066080_bb31a31f-73ae-47f6-b8cc-f3429392348f/test-curl-7491-140088652066080/0.log" Jul 19 08:40:11.896876 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:11.896841 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7491-140088652066080"] Jul 19 08:40:11.901746 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:11.901723 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7491-140088652066080"] Jul 19 08:40:11.968906 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:11.968877 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkqt5\" (UniqueName: \"kubernetes.io/projected/bb31a31f-73ae-47f6-b8cc-f3429392348f-kube-api-access-dkqt5\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:40:12.558997 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:12.558960 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb31a31f-73ae-47f6-b8cc-f3429392348f" path="/var/lib/kubelet/pods/bb31a31f-73ae-47f6-b8cc-f3429392348f/volumes" Jul 19 08:40:12.596805 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:12.596770 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:40:12.696806 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:12.696766 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:40:12.713326 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:12.713297 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7491-140088652066080" Jul 19 08:40:12.713500 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:12.713304 2574 scope.go:117] "RemoveContainer" containerID="1b40b9170c259d27080c0381b6545035da5df1cdd16dbc6b07dab948f4ded56d" Jul 19 08:40:22.589160 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:22.589114 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-72ce60-56fc9c7fb7-xfqrk"] Jul 19 08:40:22.589774 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:22.589378 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-72ce60-56fc9c7fb7-xfqrk" podUID="bcd38542-3ecd-4d77-a6dd-af4503096eca" containerName="maas-api" containerID="cri-o://8fb8f3435f3b03f03d168bc23cf523b757d33082ee1d0efe791f82c38a743f80" gracePeriod=30 Jul 19 08:40:22.622473 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:22.622441 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:40:22.769686 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:22.769582 2574 generic.go:356] "Generic (PLEG): container finished" podID="bcd38542-3ecd-4d77-a6dd-af4503096eca" containerID="8fb8f3435f3b03f03d168bc23cf523b757d33082ee1d0efe791f82c38a743f80" exitCode=0 Jul 19 08:40:22.769686 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:22.769659 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-72ce60-56fc9c7fb7-xfqrk" event={"ID":"bcd38542-3ecd-4d77-a6dd-af4503096eca","Type":"ContainerDied","Data":"8fb8f3435f3b03f03d168bc23cf523b757d33082ee1d0efe791f82c38a743f80"} Jul 19 08:40:22.862976 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:22.862947 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-72ce60-56fc9c7fb7-xfqrk" Jul 19 08:40:22.965478 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:22.965439 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qmmpz\" (UniqueName: \"kubernetes.io/projected/bcd38542-3ecd-4d77-a6dd-af4503096eca-kube-api-access-qmmpz\") pod \"bcd38542-3ecd-4d77-a6dd-af4503096eca\" (UID: \"bcd38542-3ecd-4d77-a6dd-af4503096eca\") " Jul 19 08:40:22.965478 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:22.965485 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcd38542-3ecd-4d77-a6dd-af4503096eca-openshift-service-ca\") pod \"bcd38542-3ecd-4d77-a6dd-af4503096eca\" (UID: \"bcd38542-3ecd-4d77-a6dd-af4503096eca\") " Jul 19 08:40:22.965780 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:22.965557 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcd38542-3ecd-4d77-a6dd-af4503096eca-maas-api-tls\") pod \"bcd38542-3ecd-4d77-a6dd-af4503096eca\" (UID: \"bcd38542-3ecd-4d77-a6dd-af4503096eca\") " Jul 19 08:40:22.965864 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:22.965840 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcd38542-3ecd-4d77-a6dd-af4503096eca-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "bcd38542-3ecd-4d77-a6dd-af4503096eca" (UID: "bcd38542-3ecd-4d77-a6dd-af4503096eca"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 08:40:22.967722 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:22.967692 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcd38542-3ecd-4d77-a6dd-af4503096eca-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "bcd38542-3ecd-4d77-a6dd-af4503096eca" (UID: "bcd38542-3ecd-4d77-a6dd-af4503096eca"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 08:40:22.967844 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:22.967784 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcd38542-3ecd-4d77-a6dd-af4503096eca-kube-api-access-qmmpz" (OuterVolumeSpecName: "kube-api-access-qmmpz") pod "bcd38542-3ecd-4d77-a6dd-af4503096eca" (UID: "bcd38542-3ecd-4d77-a6dd-af4503096eca"). InnerVolumeSpecName "kube-api-access-qmmpz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 08:40:23.066576 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:23.066499 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmmpz\" (UniqueName: \"kubernetes.io/projected/bcd38542-3ecd-4d77-a6dd-af4503096eca-kube-api-access-qmmpz\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:40:23.066576 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:23.066569 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcd38542-3ecd-4d77-a6dd-af4503096eca-openshift-service-ca\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:40:23.066576 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:23.066581 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcd38542-3ecd-4d77-a6dd-af4503096eca-maas-api-tls\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:40:23.777009 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:23.776975 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-72ce60-56fc9c7fb7-xfqrk" Jul 19 08:40:23.777438 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:23.777019 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-72ce60-56fc9c7fb7-xfqrk" event={"ID":"bcd38542-3ecd-4d77-a6dd-af4503096eca","Type":"ContainerDied","Data":"3259f1b10ed003151440d5c03ee7311fbda8eb0e21ab530812fdac98e10c1497"} Jul 19 08:40:23.777438 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:23.777089 2574 scope.go:117] "RemoveContainer" containerID="8fb8f3435f3b03f03d168bc23cf523b757d33082ee1d0efe791f82c38a743f80" Jul 19 08:40:23.798889 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:23.798860 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-72ce60-56fc9c7fb7-xfqrk"] Jul 19 08:40:23.802634 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:23.802607 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-72ce60-56fc9c7fb7-xfqrk"] Jul 19 08:40:24.565642 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:24.565597 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bcd38542-3ecd-4d77-a6dd-af4503096eca" path="/var/lib/kubelet/pods/bcd38542-3ecd-4d77-a6dd-af4503096eca/volumes" Jul 19 08:40:26.671381 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:26.671349 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 08:40:27.176936 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:27.176881 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:40:33.039134 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:33.039100 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:40:33.098207 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:33.098172 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:40:37.832804 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:37.832714 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-72ce60-openshift-default-7fdc94f55-24snn"] Jul 19 08:40:37.833228 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:37.833064 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-72ce60-openshift-default-7fdc94f55-24snn" podUID="6c3b1d54-8921-4761-b776-182f6c7889b1" containerName="istio-proxy" containerID="cri-o://a3e817b82e5da98b1316e332090ec8116366bffb25437bbfe901af7338e748bd" gracePeriod=30 Jul 19 08:40:37.846420 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:37.846391 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:40:38.071477 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:38.071441 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:40:38.725628 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:38.725594 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:40:38.825820 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:38.825772 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:40:43.098417 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:43.098338 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-72ce60-openshift-default-7fdc94f55-24snn" Jul 19 08:40:43.248472 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:43.248441 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c3b1d54-8921-4761-b776-182f6c7889b1-istio-podinfo\") pod \"6c3b1d54-8921-4761-b776-182f6c7889b1\" (UID: \"6c3b1d54-8921-4761-b776-182f6c7889b1\") " Jul 19 08:40:43.248679 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:43.248491 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c3b1d54-8921-4761-b776-182f6c7889b1-workload-certs\") pod \"6c3b1d54-8921-4761-b776-182f6c7889b1\" (UID: \"6c3b1d54-8921-4761-b776-182f6c7889b1\") " Jul 19 08:40:43.248679 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:43.248513 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c3b1d54-8921-4761-b776-182f6c7889b1-workload-socket\") pod \"6c3b1d54-8921-4761-b776-182f6c7889b1\" (UID: \"6c3b1d54-8921-4761-b776-182f6c7889b1\") " Jul 19 08:40:43.248679 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:43.248555 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c3b1d54-8921-4761-b776-182f6c7889b1-istiod-ca-cert\") pod \"6c3b1d54-8921-4761-b776-182f6c7889b1\" (UID: \"6c3b1d54-8921-4761-b776-182f6c7889b1\") " Jul 19 08:40:43.248679 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:43.248620 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c3b1d54-8921-4761-b776-182f6c7889b1-istio-data\") pod \"6c3b1d54-8921-4761-b776-182f6c7889b1\" (UID: \"6c3b1d54-8921-4761-b776-182f6c7889b1\") " Jul 19 08:40:43.248679 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:43.248658 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c3b1d54-8921-4761-b776-182f6c7889b1-istio-envoy\") pod \"6c3b1d54-8921-4761-b776-182f6c7889b1\" (UID: \"6c3b1d54-8921-4761-b776-182f6c7889b1\") " Jul 19 08:40:43.248950 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:43.248698 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z2lbg\" (UniqueName: \"kubernetes.io/projected/6c3b1d54-8921-4761-b776-182f6c7889b1-kube-api-access-z2lbg\") pod \"6c3b1d54-8921-4761-b776-182f6c7889b1\" (UID: \"6c3b1d54-8921-4761-b776-182f6c7889b1\") " Jul 19 08:40:43.248950 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:43.248721 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c3b1d54-8921-4761-b776-182f6c7889b1-istio-token\") pod \"6c3b1d54-8921-4761-b776-182f6c7889b1\" (UID: \"6c3b1d54-8921-4761-b776-182f6c7889b1\") " Jul 19 08:40:43.248950 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:43.248758 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c3b1d54-8921-4761-b776-182f6c7889b1-credential-socket\") pod \"6c3b1d54-8921-4761-b776-182f6c7889b1\" (UID: \"6c3b1d54-8921-4761-b776-182f6c7889b1\") " Jul 19 08:40:43.248950 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:43.248823 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c3b1d54-8921-4761-b776-182f6c7889b1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6c3b1d54-8921-4761-b776-182f6c7889b1" (UID: "6c3b1d54-8921-4761-b776-182f6c7889b1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 08:40:43.248950 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:43.248870 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c3b1d54-8921-4761-b776-182f6c7889b1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6c3b1d54-8921-4761-b776-182f6c7889b1" (UID: "6c3b1d54-8921-4761-b776-182f6c7889b1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 08:40:43.249222 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:43.249085 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c3b1d54-8921-4761-b776-182f6c7889b1-workload-certs\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:40:43.249222 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:43.249106 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c3b1d54-8921-4761-b776-182f6c7889b1-workload-socket\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:40:43.249222 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:43.249117 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c3b1d54-8921-4761-b776-182f6c7889b1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6c3b1d54-8921-4761-b776-182f6c7889b1" (UID: "6c3b1d54-8921-4761-b776-182f6c7889b1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 08:40:43.249376 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:43.249245 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c3b1d54-8921-4761-b776-182f6c7889b1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6c3b1d54-8921-4761-b776-182f6c7889b1" (UID: "6c3b1d54-8921-4761-b776-182f6c7889b1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 08:40:43.249592 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:43.249569 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c3b1d54-8921-4761-b776-182f6c7889b1-istio-data" (OuterVolumeSpecName: "istio-data") pod "6c3b1d54-8921-4761-b776-182f6c7889b1" (UID: "6c3b1d54-8921-4761-b776-182f6c7889b1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 08:40:43.250969 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:43.250930 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6c3b1d54-8921-4761-b776-182f6c7889b1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6c3b1d54-8921-4761-b776-182f6c7889b1" (UID: "6c3b1d54-8921-4761-b776-182f6c7889b1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 19 08:40:43.251060 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:43.250987 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c3b1d54-8921-4761-b776-182f6c7889b1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6c3b1d54-8921-4761-b776-182f6c7889b1" (UID: "6c3b1d54-8921-4761-b776-182f6c7889b1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 08:40:43.251060 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:43.251023 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c3b1d54-8921-4761-b776-182f6c7889b1-istio-token" (OuterVolumeSpecName: "istio-token") pod "6c3b1d54-8921-4761-b776-182f6c7889b1" (UID: "6c3b1d54-8921-4761-b776-182f6c7889b1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 08:40:43.251060 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:43.251045 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c3b1d54-8921-4761-b776-182f6c7889b1-kube-api-access-z2lbg" (OuterVolumeSpecName: "kube-api-access-z2lbg") pod "6c3b1d54-8921-4761-b776-182f6c7889b1" (UID: "6c3b1d54-8921-4761-b776-182f6c7889b1"). InnerVolumeSpecName "kube-api-access-z2lbg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 08:40:43.350427 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:43.350343 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2lbg\" (UniqueName: \"kubernetes.io/projected/6c3b1d54-8921-4761-b776-182f6c7889b1-kube-api-access-z2lbg\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:40:43.350427 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:43.350373 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c3b1d54-8921-4761-b776-182f6c7889b1-istio-token\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:40:43.350427 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:43.350383 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c3b1d54-8921-4761-b776-182f6c7889b1-credential-socket\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:40:43.350427 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:43.350392 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c3b1d54-8921-4761-b776-182f6c7889b1-istio-podinfo\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:40:43.350427 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:43.350401 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c3b1d54-8921-4761-b776-182f6c7889b1-istiod-ca-cert\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:40:43.350427 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:43.350408 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c3b1d54-8921-4761-b776-182f6c7889b1-istio-data\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:40:43.350427 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:43.350416 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c3b1d54-8921-4761-b776-182f6c7889b1-istio-envoy\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:40:43.895197 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:43.895164 2574 generic.go:356] "Generic (PLEG): container finished" podID="6c3b1d54-8921-4761-b776-182f6c7889b1" containerID="a3e817b82e5da98b1316e332090ec8116366bffb25437bbfe901af7338e748bd" exitCode=0 Jul 19 08:40:43.895381 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:43.895216 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-72ce60-openshift-default-7fdc94f55-24snn" Jul 19 08:40:43.895381 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:43.895237 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-72ce60-openshift-default-7fdc94f55-24snn" event={"ID":"6c3b1d54-8921-4761-b776-182f6c7889b1","Type":"ContainerDied","Data":"a3e817b82e5da98b1316e332090ec8116366bffb25437bbfe901af7338e748bd"} Jul 19 08:40:43.895381 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:43.895316 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-72ce60-openshift-default-7fdc94f55-24snn" event={"ID":"6c3b1d54-8921-4761-b776-182f6c7889b1","Type":"ContainerDied","Data":"56cd13d0a820494ac2ad565097636f7a7b64411153c361022675a116ae61f288"} Jul 19 08:40:43.895381 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:43.895338 2574 scope.go:117] "RemoveContainer" containerID="a3e817b82e5da98b1316e332090ec8116366bffb25437bbfe901af7338e748bd" Jul 19 08:40:43.905295 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:43.905277 2574 scope.go:117] "RemoveContainer" containerID="66a22f07cd26556bb17a657578b5fb7f2ac38eeed2323099571e4ffb271caf7f" Jul 19 08:40:43.913394 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:43.913376 2574 scope.go:117] "RemoveContainer" containerID="a3e817b82e5da98b1316e332090ec8116366bffb25437bbfe901af7338e748bd" Jul 19 08:40:43.913658 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:40:43.913637 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3e817b82e5da98b1316e332090ec8116366bffb25437bbfe901af7338e748bd\": container with ID starting with a3e817b82e5da98b1316e332090ec8116366bffb25437bbfe901af7338e748bd not found: ID does not exist" containerID="a3e817b82e5da98b1316e332090ec8116366bffb25437bbfe901af7338e748bd" Jul 19 08:40:43.913711 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:43.913668 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3e817b82e5da98b1316e332090ec8116366bffb25437bbfe901af7338e748bd"} err="failed to get container status \"a3e817b82e5da98b1316e332090ec8116366bffb25437bbfe901af7338e748bd\": rpc error: code = NotFound desc = could not find container \"a3e817b82e5da98b1316e332090ec8116366bffb25437bbfe901af7338e748bd\": container with ID starting with a3e817b82e5da98b1316e332090ec8116366bffb25437bbfe901af7338e748bd not found: ID does not exist" Jul 19 08:40:43.913711 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:43.913686 2574 scope.go:117] "RemoveContainer" containerID="66a22f07cd26556bb17a657578b5fb7f2ac38eeed2323099571e4ffb271caf7f" Jul 19 08:40:43.913979 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:40:43.913960 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66a22f07cd26556bb17a657578b5fb7f2ac38eeed2323099571e4ffb271caf7f\": container with ID starting with 66a22f07cd26556bb17a657578b5fb7f2ac38eeed2323099571e4ffb271caf7f not found: ID does not exist" containerID="66a22f07cd26556bb17a657578b5fb7f2ac38eeed2323099571e4ffb271caf7f" Jul 19 08:40:43.914044 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:43.913988 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66a22f07cd26556bb17a657578b5fb7f2ac38eeed2323099571e4ffb271caf7f"} err="failed to get container status \"66a22f07cd26556bb17a657578b5fb7f2ac38eeed2323099571e4ffb271caf7f\": rpc error: code = NotFound desc = could not find container \"66a22f07cd26556bb17a657578b5fb7f2ac38eeed2323099571e4ffb271caf7f\": container with ID starting with 66a22f07cd26556bb17a657578b5fb7f2ac38eeed2323099571e4ffb271caf7f not found: ID does not exist" Jul 19 08:40:43.917083 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:43.917047 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-72ce60-openshift-default-7fdc94f55-24snn"] Jul 19 08:40:43.920430 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:43.920407 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-72ce60-openshift-default-7fdc94f55-24snn"] Jul 19 08:40:44.558537 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:44.558494 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c3b1d54-8921-4761-b776-182f6c7889b1" path="/var/lib/kubelet/pods/6c3b1d54-8921-4761-b776-182f6c7889b1/volumes" Jul 19 08:40:48.660158 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:48.660123 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:40:51.877256 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:51.873997 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tgs9r"] Jul 19 08:40:53.283298 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:53.283259 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:40:59.114234 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:59.114199 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:40:59.175253 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:40:59.175220 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:41:03.944310 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:03.944264 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-190fde-openshift-default-5c6c4d7755-8dlqr"] Jul 19 08:41:03.944875 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:03.944846 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-190fde-openshift-default-5c6c4d7755-8dlqr" podUID="81baf455-fff0-45eb-aaf0-ee32450d271b" containerName="istio-proxy" containerID="cri-o://4d234c9700e4eec23ca8787bc2aa7bb4710593c0bae08422cb41800b89cf68a8" gracePeriod=30 Jul 19 08:41:03.958922 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:03.958878 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:41:04.190671 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:04.190627 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:41:04.585588 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:04.585555 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-54b5cb644d-775fd_7e666b4f-2534-4d09-889e-a23a63ecef62/authorino/0.log" Jul 19 08:41:09.199850 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:09.199820 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-190fde-openshift-default-5c6c4d7755-8dlqr" Jul 19 08:41:09.295961 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:09.295865 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81baf455-fff0-45eb-aaf0-ee32450d271b-istiod-ca-cert\") pod \"81baf455-fff0-45eb-aaf0-ee32450d271b\" (UID: \"81baf455-fff0-45eb-aaf0-ee32450d271b\") " Jul 19 08:41:09.295961 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:09.295936 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81baf455-fff0-45eb-aaf0-ee32450d271b-istio-data\") pod \"81baf455-fff0-45eb-aaf0-ee32450d271b\" (UID: \"81baf455-fff0-45eb-aaf0-ee32450d271b\") " Jul 19 08:41:09.296206 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:09.295965 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k8bpl\" (UniqueName: \"kubernetes.io/projected/81baf455-fff0-45eb-aaf0-ee32450d271b-kube-api-access-k8bpl\") pod \"81baf455-fff0-45eb-aaf0-ee32450d271b\" (UID: \"81baf455-fff0-45eb-aaf0-ee32450d271b\") " Jul 19 08:41:09.296206 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:09.295993 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81baf455-fff0-45eb-aaf0-ee32450d271b-istio-envoy\") pod \"81baf455-fff0-45eb-aaf0-ee32450d271b\" (UID: \"81baf455-fff0-45eb-aaf0-ee32450d271b\") " Jul 19 08:41:09.296206 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:09.296033 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81baf455-fff0-45eb-aaf0-ee32450d271b-workload-certs\") pod \"81baf455-fff0-45eb-aaf0-ee32450d271b\" (UID: \"81baf455-fff0-45eb-aaf0-ee32450d271b\") " Jul 19 08:41:09.296206 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:09.296063 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81baf455-fff0-45eb-aaf0-ee32450d271b-istio-token\") pod \"81baf455-fff0-45eb-aaf0-ee32450d271b\" (UID: \"81baf455-fff0-45eb-aaf0-ee32450d271b\") " Jul 19 08:41:09.296206 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:09.296097 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81baf455-fff0-45eb-aaf0-ee32450d271b-workload-socket\") pod \"81baf455-fff0-45eb-aaf0-ee32450d271b\" (UID: \"81baf455-fff0-45eb-aaf0-ee32450d271b\") " Jul 19 08:41:09.296206 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:09.296160 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81baf455-fff0-45eb-aaf0-ee32450d271b-credential-socket\") pod \"81baf455-fff0-45eb-aaf0-ee32450d271b\" (UID: \"81baf455-fff0-45eb-aaf0-ee32450d271b\") " Jul 19 08:41:09.296206 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:09.296187 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81baf455-fff0-45eb-aaf0-ee32450d271b-istio-podinfo\") pod \"81baf455-fff0-45eb-aaf0-ee32450d271b\" (UID: \"81baf455-fff0-45eb-aaf0-ee32450d271b\") " Jul 19 08:41:09.296635 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:09.296317 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81baf455-fff0-45eb-aaf0-ee32450d271b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "81baf455-fff0-45eb-aaf0-ee32450d271b" (UID: "81baf455-fff0-45eb-aaf0-ee32450d271b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 08:41:09.296635 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:09.296428 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81baf455-fff0-45eb-aaf0-ee32450d271b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "81baf455-fff0-45eb-aaf0-ee32450d271b" (UID: "81baf455-fff0-45eb-aaf0-ee32450d271b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 08:41:09.296635 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:09.296586 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81baf455-fff0-45eb-aaf0-ee32450d271b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "81baf455-fff0-45eb-aaf0-ee32450d271b" (UID: "81baf455-fff0-45eb-aaf0-ee32450d271b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 08:41:09.296803 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:09.296634 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81baf455-fff0-45eb-aaf0-ee32450d271b-istiod-ca-cert\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:41:09.296803 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:09.296653 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81baf455-fff0-45eb-aaf0-ee32450d271b-workload-socket\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:41:09.296916 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:09.296843 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81baf455-fff0-45eb-aaf0-ee32450d271b-istio-data" (OuterVolumeSpecName: "istio-data") pod "81baf455-fff0-45eb-aaf0-ee32450d271b" (UID: "81baf455-fff0-45eb-aaf0-ee32450d271b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 08:41:09.296980 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:09.296941 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81baf455-fff0-45eb-aaf0-ee32450d271b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "81baf455-fff0-45eb-aaf0-ee32450d271b" (UID: "81baf455-fff0-45eb-aaf0-ee32450d271b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 08:41:09.298422 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:09.298398 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81baf455-fff0-45eb-aaf0-ee32450d271b-istio-token" (OuterVolumeSpecName: "istio-token") pod "81baf455-fff0-45eb-aaf0-ee32450d271b" (UID: "81baf455-fff0-45eb-aaf0-ee32450d271b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 08:41:09.298422 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:09.298409 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81baf455-fff0-45eb-aaf0-ee32450d271b-kube-api-access-k8bpl" (OuterVolumeSpecName: "kube-api-access-k8bpl") pod "81baf455-fff0-45eb-aaf0-ee32450d271b" (UID: "81baf455-fff0-45eb-aaf0-ee32450d271b"). InnerVolumeSpecName "kube-api-access-k8bpl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 08:41:09.298830 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:09.298807 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81baf455-fff0-45eb-aaf0-ee32450d271b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "81baf455-fff0-45eb-aaf0-ee32450d271b" (UID: "81baf455-fff0-45eb-aaf0-ee32450d271b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 08:41:09.298908 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:09.298813 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/81baf455-fff0-45eb-aaf0-ee32450d271b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "81baf455-fff0-45eb-aaf0-ee32450d271b" (UID: "81baf455-fff0-45eb-aaf0-ee32450d271b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 19 08:41:09.397301 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:09.397260 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81baf455-fff0-45eb-aaf0-ee32450d271b-istio-data\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:41:09.397301 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:09.397294 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8bpl\" (UniqueName: \"kubernetes.io/projected/81baf455-fff0-45eb-aaf0-ee32450d271b-kube-api-access-k8bpl\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:41:09.397301 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:09.397304 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81baf455-fff0-45eb-aaf0-ee32450d271b-istio-envoy\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:41:09.397610 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:09.397313 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81baf455-fff0-45eb-aaf0-ee32450d271b-workload-certs\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:41:09.397610 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:09.397324 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81baf455-fff0-45eb-aaf0-ee32450d271b-istio-token\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:41:09.397610 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:09.397331 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81baf455-fff0-45eb-aaf0-ee32450d271b-credential-socket\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:41:09.397610 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:09.397341 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81baf455-fff0-45eb-aaf0-ee32450d271b-istio-podinfo\") on node \"ip-10-0-136-70.ec2.internal\" DevicePath \"\"" Jul 19 08:41:09.888588 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:09.888556 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-gdc7l_33b278bc-869f-4d53-9c01-78e3252c1fc8/manager/0.log" Jul 19 08:41:10.041501 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:10.041467 2574 generic.go:356] "Generic (PLEG): container finished" podID="81baf455-fff0-45eb-aaf0-ee32450d271b" containerID="4d234c9700e4eec23ca8787bc2aa7bb4710593c0bae08422cb41800b89cf68a8" exitCode=0 Jul 19 08:41:10.041710 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:10.041541 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-190fde-openshift-default-5c6c4d7755-8dlqr" Jul 19 08:41:10.041710 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:10.041560 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-190fde-openshift-default-5c6c4d7755-8dlqr" event={"ID":"81baf455-fff0-45eb-aaf0-ee32450d271b","Type":"ContainerDied","Data":"4d234c9700e4eec23ca8787bc2aa7bb4710593c0bae08422cb41800b89cf68a8"} Jul 19 08:41:10.041710 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:10.041615 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-190fde-openshift-default-5c6c4d7755-8dlqr" event={"ID":"81baf455-fff0-45eb-aaf0-ee32450d271b","Type":"ContainerDied","Data":"87dc151194ee7e9f067bb60c5fc67d30e5b13252d31c4b8014110f734a7a37eb"} Jul 19 08:41:10.041710 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:10.041636 2574 scope.go:117] "RemoveContainer" containerID="4d234c9700e4eec23ca8787bc2aa7bb4710593c0bae08422cb41800b89cf68a8" Jul 19 08:41:10.051700 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:10.051676 2574 scope.go:117] "RemoveContainer" containerID="769a546ce6f3cd95a34565cd9a64ef428b4d6978960c9b25dfcdf8edc8ba2992" Jul 19 08:41:10.060676 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:10.060503 2574 scope.go:117] "RemoveContainer" containerID="4d234c9700e4eec23ca8787bc2aa7bb4710593c0bae08422cb41800b89cf68a8" Jul 19 08:41:10.060767 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:41:10.060748 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d234c9700e4eec23ca8787bc2aa7bb4710593c0bae08422cb41800b89cf68a8\": container with ID starting with 4d234c9700e4eec23ca8787bc2aa7bb4710593c0bae08422cb41800b89cf68a8 not found: ID does not exist" containerID="4d234c9700e4eec23ca8787bc2aa7bb4710593c0bae08422cb41800b89cf68a8" Jul 19 08:41:10.060807 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:10.060776 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d234c9700e4eec23ca8787bc2aa7bb4710593c0bae08422cb41800b89cf68a8"} err="failed to get container status \"4d234c9700e4eec23ca8787bc2aa7bb4710593c0bae08422cb41800b89cf68a8\": rpc error: code = NotFound desc = could not find container \"4d234c9700e4eec23ca8787bc2aa7bb4710593c0bae08422cb41800b89cf68a8\": container with ID starting with 4d234c9700e4eec23ca8787bc2aa7bb4710593c0bae08422cb41800b89cf68a8 not found: ID does not exist" Jul 19 08:41:10.060807 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:10.060791 2574 scope.go:117] "RemoveContainer" containerID="769a546ce6f3cd95a34565cd9a64ef428b4d6978960c9b25dfcdf8edc8ba2992" Jul 19 08:41:10.061005 ip-10-0-136-70 kubenswrapper[2574]: E0719 08:41:10.060989 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"769a546ce6f3cd95a34565cd9a64ef428b4d6978960c9b25dfcdf8edc8ba2992\": container with ID starting with 769a546ce6f3cd95a34565cd9a64ef428b4d6978960c9b25dfcdf8edc8ba2992 not found: ID does not exist" containerID="769a546ce6f3cd95a34565cd9a64ef428b4d6978960c9b25dfcdf8edc8ba2992" Jul 19 08:41:10.061056 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:10.061010 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"769a546ce6f3cd95a34565cd9a64ef428b4d6978960c9b25dfcdf8edc8ba2992"} err="failed to get container status \"769a546ce6f3cd95a34565cd9a64ef428b4d6978960c9b25dfcdf8edc8ba2992\": rpc error: code = NotFound desc = could not find container \"769a546ce6f3cd95a34565cd9a64ef428b4d6978960c9b25dfcdf8edc8ba2992\": container with ID starting with 769a546ce6f3cd95a34565cd9a64ef428b4d6978960c9b25dfcdf8edc8ba2992 not found: ID does not exist" Jul 19 08:41:10.065034 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:10.065012 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-190fde-openshift-default-5c6c4d7755-8dlqr"] Jul 19 08:41:10.068697 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:10.068677 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-190fde-openshift-default-5c6c4d7755-8dlqr"] Jul 19 08:41:10.110741 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:10.110719 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-lvdn2_a402f5d0-4bcc-45be-9d79-13c7e577d3c3/server/0.log" Jul 19 08:41:10.218938 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:10.218906 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-85fwf_d389a635-9a6d-42d5-b70c-b78a0898b929/manager/0.log" Jul 19 08:41:10.449585 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:10.449552 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-789467ff8c-j7mlr_6e217fee-5171-40e1-82fb-155e4a0431d1/copy-manifests/0.log" Jul 19 08:41:10.461347 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:10.461324 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-789467ff8c-j7mlr_6e217fee-5171-40e1-82fb-155e4a0431d1/manager/0.log" Jul 19 08:41:10.558302 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:10.558218 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81baf455-fff0-45eb-aaf0-ee32450d271b" path="/var/lib/kubelet/pods/81baf455-fff0-45eb-aaf0-ee32450d271b/volumes" Jul 19 08:41:11.057048 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:11.057010 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29740800-6qc4k_fa862c8e-ccc2-4863-8edd-0430e652a9cc/cleanup/0.log" Jul 19 08:41:13.663329 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:13.663294 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-54b5cb644d-775fd_7e666b4f-2534-4d09-889e-a23a63ecef62/authorino/0.log" Jul 19 08:41:13.775385 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:13.775340 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-7f54b_13322af6-4caf-4e03-89fe-4491af5d4255/manager/0.log" Jul 19 08:41:13.885832 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:13.885800 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-ghjrd_727df024-63b4-4a1e-81a6-b546c6ebda2d/manager/0.log" Jul 19 08:41:14.106595 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:14.106556 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-ttzxt_2b0ebf15-5daf-47dd-8acb-3a668927d267/registry-server/0.log" Jul 19 08:41:14.216332 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:14.216299 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-z4jm9_80be3ae2-48da-4746-91f7-a32946f64b04/manager/0.log" Jul 19 08:41:14.327406 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:14.327374 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-tgs9r_78f72907-2f0e-487c-b31e-70e2a8727400/limitador/0.log" Jul 19 08:41:14.441468 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:14.441387 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-ssftg_df18da6e-66ff-40d0-9957-3e82b79e14d4/manager/0.log" Jul 19 08:41:15.113115 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:15.113085 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-56f59f876d-qqxn9_1c5887e0-25ef-48da-a06c-a1874a64e2c7/kube-auth-proxy/0.log" Jul 19 08:41:15.558351 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:15.558322 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5669bd84dd-j7hs7_f56affaf-b38d-4ea4-ac8c-309edaa8852e/router/0.log" Jul 19 08:41:21.440806 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:21.440776 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9nlzs/must-gather-d2gtq"] Jul 19 08:41:21.441185 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:21.441149 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c3b1d54-8921-4761-b776-182f6c7889b1" containerName="istio-proxy" Jul 19 08:41:21.441185 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:21.441161 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c3b1d54-8921-4761-b776-182f6c7889b1" containerName="istio-proxy" Jul 19 08:41:21.441185 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:21.441169 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81baf455-fff0-45eb-aaf0-ee32450d271b" containerName="istio-proxy" Jul 19 08:41:21.441185 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:21.441175 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="81baf455-fff0-45eb-aaf0-ee32450d271b" containerName="istio-proxy" Jul 19 08:41:21.441324 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:21.441189 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb31a31f-73ae-47f6-b8cc-f3429392348f" containerName="test-curl-7491-140088652066080" Jul 19 08:41:21.441324 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:21.441196 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb31a31f-73ae-47f6-b8cc-f3429392348f" containerName="test-curl-7491-140088652066080" Jul 19 08:41:21.441324 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:21.441207 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81baf455-fff0-45eb-aaf0-ee32450d271b" containerName="istio-proxy" Jul 19 08:41:21.441324 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:21.441213 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="81baf455-fff0-45eb-aaf0-ee32450d271b" containerName="istio-proxy" Jul 19 08:41:21.441324 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:21.441228 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bcd38542-3ecd-4d77-a6dd-af4503096eca" containerName="maas-api" Jul 19 08:41:21.441324 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:21.441233 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="bcd38542-3ecd-4d77-a6dd-af4503096eca" containerName="maas-api" Jul 19 08:41:21.441324 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:21.441284 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="81baf455-fff0-45eb-aaf0-ee32450d271b" containerName="istio-proxy" Jul 19 08:41:21.441324 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:21.441292 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="bcd38542-3ecd-4d77-a6dd-af4503096eca" containerName="maas-api" Jul 19 08:41:21.441324 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:21.441299 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c3b1d54-8921-4761-b776-182f6c7889b1" containerName="istio-proxy" Jul 19 08:41:21.441324 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:21.441307 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c3b1d54-8921-4761-b776-182f6c7889b1" containerName="istio-proxy" Jul 19 08:41:21.441324 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:21.441313 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb31a31f-73ae-47f6-b8cc-f3429392348f" containerName="test-curl-7491-140088652066080" Jul 19 08:41:21.441712 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:21.441365 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c3b1d54-8921-4761-b776-182f6c7889b1" containerName="istio-proxy" Jul 19 08:41:21.441712 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:21.441370 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c3b1d54-8921-4761-b776-182f6c7889b1" containerName="istio-proxy" Jul 19 08:41:21.441712 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:21.441441 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="81baf455-fff0-45eb-aaf0-ee32450d271b" containerName="istio-proxy" Jul 19 08:41:21.446102 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:21.446074 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9nlzs/must-gather-d2gtq" Jul 19 08:41:21.448765 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:21.448737 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9nlzs\"/\"kube-root-ca.crt\"" Jul 19 08:41:21.448765 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:21.448748 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9nlzs\"/\"openshift-service-ca.crt\"" Jul 19 08:41:21.453120 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:21.453082 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9nlzs/must-gather-d2gtq"] Jul 19 08:41:21.456306 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:21.456275 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:41:21.504611 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:21.504572 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3fa7b92b-e419-4bd6-82cb-4689202cbdb7-must-gather-output\") pod \"must-gather-d2gtq\" (UID: \"3fa7b92b-e419-4bd6-82cb-4689202cbdb7\") " pod="openshift-must-gather-9nlzs/must-gather-d2gtq" Jul 19 08:41:21.504813 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:21.504626 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcxvg\" (UniqueName: \"kubernetes.io/projected/3fa7b92b-e419-4bd6-82cb-4689202cbdb7-kube-api-access-wcxvg\") pod \"must-gather-d2gtq\" (UID: \"3fa7b92b-e419-4bd6-82cb-4689202cbdb7\") " pod="openshift-must-gather-9nlzs/must-gather-d2gtq" Jul 19 08:41:21.550030 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:21.549997 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:41:21.605989 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:21.605949 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3fa7b92b-e419-4bd6-82cb-4689202cbdb7-must-gather-output\") pod \"must-gather-d2gtq\" (UID: \"3fa7b92b-e419-4bd6-82cb-4689202cbdb7\") " pod="openshift-must-gather-9nlzs/must-gather-d2gtq" Jul 19 08:41:21.606213 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:21.606004 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcxvg\" (UniqueName: \"kubernetes.io/projected/3fa7b92b-e419-4bd6-82cb-4689202cbdb7-kube-api-access-wcxvg\") pod \"must-gather-d2gtq\" (UID: \"3fa7b92b-e419-4bd6-82cb-4689202cbdb7\") " pod="openshift-must-gather-9nlzs/must-gather-d2gtq" Jul 19 08:41:21.606294 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:21.606274 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3fa7b92b-e419-4bd6-82cb-4689202cbdb7-must-gather-output\") pod \"must-gather-d2gtq\" (UID: \"3fa7b92b-e419-4bd6-82cb-4689202cbdb7\") " pod="openshift-must-gather-9nlzs/must-gather-d2gtq" Jul 19 08:41:21.614177 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:21.614141 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcxvg\" (UniqueName: \"kubernetes.io/projected/3fa7b92b-e419-4bd6-82cb-4689202cbdb7-kube-api-access-wcxvg\") pod \"must-gather-d2gtq\" (UID: \"3fa7b92b-e419-4bd6-82cb-4689202cbdb7\") " pod="openshift-must-gather-9nlzs/must-gather-d2gtq" Jul 19 08:41:21.756617 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:21.756582 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9nlzs/must-gather-d2gtq" Jul 19 08:41:21.892671 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:21.892647 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9nlzs/must-gather-d2gtq"] Jul 19 08:41:21.894862 ip-10-0-136-70 kubenswrapper[2574]: W0719 08:41:21.894832 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3fa7b92b_e419_4bd6_82cb_4689202cbdb7.slice/crio-c11943da60e749db85edd4de0bc057f5a799a0ff7db7abc6fd057bc847bd550b WatchSource:0}: Error finding container c11943da60e749db85edd4de0bc057f5a799a0ff7db7abc6fd057bc847bd550b: Status 404 returned error can't find the container with id c11943da60e749db85edd4de0bc057f5a799a0ff7db7abc6fd057bc847bd550b Jul 19 08:41:22.101508 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:22.101409 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9nlzs/must-gather-d2gtq" event={"ID":"3fa7b92b-e419-4bd6-82cb-4689202cbdb7","Type":"ContainerStarted","Data":"c11943da60e749db85edd4de0bc057f5a799a0ff7db7abc6fd057bc847bd550b"} Jul 19 08:41:23.111781 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:23.111017 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9nlzs/must-gather-d2gtq" event={"ID":"3fa7b92b-e419-4bd6-82cb-4689202cbdb7","Type":"ContainerStarted","Data":"c1a365bece87b88ebf4f7d69eb876e7320889dc3302775bdacc7b286eeb5e733"} Jul 19 08:41:23.111781 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:23.111074 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9nlzs/must-gather-d2gtq" event={"ID":"3fa7b92b-e419-4bd6-82cb-4689202cbdb7","Type":"ContainerStarted","Data":"67190248cf04ce6a3cd7d20743c37b866a88832851ef9d9bde75f3cac03f7c3d"} Jul 19 08:41:23.136039 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:23.135960 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-9nlzs/must-gather-d2gtq" podStartSLOduration=1.336127554 podStartE2EDuration="2.135939524s" totalImagesPullingTime="799.81197ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 08:41:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:41:23.126219973 +0000 UTC m=+3549.480529573" watchObservedRunningTime="2026-07-19 08:41:23.135939524 +0000 UTC m=+3549.490249128" Jul 19 08:41:24.222866 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:24.222834 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-p5wwk_17c9482e-aaa4-41b0-8aad-ac6f428f1f48/global-pull-secret-syncer/0.log" Jul 19 08:41:24.306574 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:24.306533 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" Jul 19 08:41:24.446200 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:24.446169 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-70.ec2.internal_76b974fafc626c9019b0378af47d7689/haproxy/0.log" Jul 19 08:41:30.288610 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:30.288573 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-54b5cb644d-775fd_7e666b4f-2534-4d09-889e-a23a63ecef62/authorino/0.log" Jul 19 08:41:30.330756 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:30.330724 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-7f54b_13322af6-4caf-4e03-89fe-4491af5d4255/manager/0.log" Jul 19 08:41:30.359253 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:30.359042 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-ghjrd_727df024-63b4-4a1e-81a6-b546c6ebda2d/manager/0.log" Jul 19 08:41:30.439197 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:30.439062 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-ttzxt_2b0ebf15-5daf-47dd-8acb-3a668927d267/registry-server/0.log" Jul 19 08:41:30.615593 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:30.615079 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-z4jm9_80be3ae2-48da-4746-91f7-a32946f64b04/manager/0.log" Jul 19 08:41:30.636561 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:30.636464 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-tgs9r_78f72907-2f0e-487c-b31e-70e2a8727400/limitador/0.log" Jul 19 08:41:30.708562 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:30.708514 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-ssftg_df18da6e-66ff-40d0-9957-3e82b79e14d4/manager/0.log" Jul 19 08:41:32.640888 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:32.640800 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-5sgkp_fbd767a8-d8bc-4f92-a449-46474f1ac52d/cluster-monitoring-operator/0.log" Jul 19 08:41:32.868230 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:32.868084 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-g6cp9_762f12f8-e846-45b9-9ba5-aafbb2aabb66/node-exporter/0.log" Jul 19 08:41:32.917247 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:32.917152 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-g6cp9_762f12f8-e846-45b9-9ba5-aafbb2aabb66/kube-rbac-proxy/0.log" Jul 19 08:41:32.934628 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:32.934589 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-g6cp9_762f12f8-e846-45b9-9ba5-aafbb2aabb66/init-textfile/0.log" Jul 19 08:41:34.458665 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:34.458623 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9nlzs/perf-node-gather-daemonset-p72vg"] Jul 19 08:41:34.465297 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:34.465264 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9nlzs/perf-node-gather-daemonset-p72vg" Jul 19 08:41:34.469859 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:34.469833 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-9nlzs\"/\"default-dockercfg-5jrnw\"" Jul 19 08:41:34.472568 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:34.472537 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9nlzs/perf-node-gather-daemonset-p72vg"] Jul 19 08:41:34.551362 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:34.551322 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9ef09a6d-8f13-4f49-801e-6d29053f187e-lib-modules\") pod \"perf-node-gather-daemonset-p72vg\" (UID: \"9ef09a6d-8f13-4f49-801e-6d29053f187e\") " pod="openshift-must-gather-9nlzs/perf-node-gather-daemonset-p72vg" Jul 19 08:41:34.551362 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:34.551372 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s265b\" (UniqueName: \"kubernetes.io/projected/9ef09a6d-8f13-4f49-801e-6d29053f187e-kube-api-access-s265b\") pod \"perf-node-gather-daemonset-p72vg\" (UID: \"9ef09a6d-8f13-4f49-801e-6d29053f187e\") " pod="openshift-must-gather-9nlzs/perf-node-gather-daemonset-p72vg" Jul 19 08:41:34.551653 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:34.551410 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9ef09a6d-8f13-4f49-801e-6d29053f187e-proc\") pod \"perf-node-gather-daemonset-p72vg\" (UID: \"9ef09a6d-8f13-4f49-801e-6d29053f187e\") " pod="openshift-must-gather-9nlzs/perf-node-gather-daemonset-p72vg" Jul 19 08:41:34.551653 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:34.551431 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9ef09a6d-8f13-4f49-801e-6d29053f187e-podres\") pod \"perf-node-gather-daemonset-p72vg\" (UID: \"9ef09a6d-8f13-4f49-801e-6d29053f187e\") " pod="openshift-must-gather-9nlzs/perf-node-gather-daemonset-p72vg" Jul 19 08:41:34.551653 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:34.551461 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9ef09a6d-8f13-4f49-801e-6d29053f187e-sys\") pod \"perf-node-gather-daemonset-p72vg\" (UID: \"9ef09a6d-8f13-4f49-801e-6d29053f187e\") " pod="openshift-must-gather-9nlzs/perf-node-gather-daemonset-p72vg" Jul 19 08:41:34.630065 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:34.630028 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-7bc7489cb5-8w7nz_a4349d67-121d-44bb-9d73-711ee0465237/networking-console-plugin/0.log" Jul 19 08:41:34.652616 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:34.652572 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9ef09a6d-8f13-4f49-801e-6d29053f187e-proc\") pod \"perf-node-gather-daemonset-p72vg\" (UID: \"9ef09a6d-8f13-4f49-801e-6d29053f187e\") " pod="openshift-must-gather-9nlzs/perf-node-gather-daemonset-p72vg" Jul 19 08:41:34.652874 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:34.652854 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9ef09a6d-8f13-4f49-801e-6d29053f187e-podres\") pod \"perf-node-gather-daemonset-p72vg\" (UID: \"9ef09a6d-8f13-4f49-801e-6d29053f187e\") " pod="openshift-must-gather-9nlzs/perf-node-gather-daemonset-p72vg" Jul 19 08:41:34.653024 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:34.653009 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9ef09a6d-8f13-4f49-801e-6d29053f187e-sys\") pod \"perf-node-gather-daemonset-p72vg\" (UID: \"9ef09a6d-8f13-4f49-801e-6d29053f187e\") " pod="openshift-must-gather-9nlzs/perf-node-gather-daemonset-p72vg" Jul 19 08:41:34.653245 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:34.653231 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9ef09a6d-8f13-4f49-801e-6d29053f187e-lib-modules\") pod \"perf-node-gather-daemonset-p72vg\" (UID: \"9ef09a6d-8f13-4f49-801e-6d29053f187e\") " pod="openshift-must-gather-9nlzs/perf-node-gather-daemonset-p72vg" Jul 19 08:41:34.653378 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:34.653361 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s265b\" (UniqueName: \"kubernetes.io/projected/9ef09a6d-8f13-4f49-801e-6d29053f187e-kube-api-access-s265b\") pod \"perf-node-gather-daemonset-p72vg\" (UID: \"9ef09a6d-8f13-4f49-801e-6d29053f187e\") " pod="openshift-must-gather-9nlzs/perf-node-gather-daemonset-p72vg" Jul 19 08:41:34.653961 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:34.653939 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9ef09a6d-8f13-4f49-801e-6d29053f187e-proc\") pod \"perf-node-gather-daemonset-p72vg\" (UID: \"9ef09a6d-8f13-4f49-801e-6d29053f187e\") " pod="openshift-must-gather-9nlzs/perf-node-gather-daemonset-p72vg" Jul 19 08:41:34.654244 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:34.654228 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9ef09a6d-8f13-4f49-801e-6d29053f187e-podres\") pod \"perf-node-gather-daemonset-p72vg\" (UID: \"9ef09a6d-8f13-4f49-801e-6d29053f187e\") " pod="openshift-must-gather-9nlzs/perf-node-gather-daemonset-p72vg" Jul 19 08:41:34.654409 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:34.654388 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9ef09a6d-8f13-4f49-801e-6d29053f187e-sys\") pod \"perf-node-gather-daemonset-p72vg\" (UID: \"9ef09a6d-8f13-4f49-801e-6d29053f187e\") " pod="openshift-must-gather-9nlzs/perf-node-gather-daemonset-p72vg" Jul 19 08:41:34.654642 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:34.654614 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9ef09a6d-8f13-4f49-801e-6d29053f187e-lib-modules\") pod \"perf-node-gather-daemonset-p72vg\" (UID: \"9ef09a6d-8f13-4f49-801e-6d29053f187e\") " pod="openshift-must-gather-9nlzs/perf-node-gather-daemonset-p72vg" Jul 19 08:41:34.661986 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:34.661959 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s265b\" (UniqueName: \"kubernetes.io/projected/9ef09a6d-8f13-4f49-801e-6d29053f187e-kube-api-access-s265b\") pod \"perf-node-gather-daemonset-p72vg\" (UID: \"9ef09a6d-8f13-4f49-801e-6d29053f187e\") " pod="openshift-must-gather-9nlzs/perf-node-gather-daemonset-p72vg" Jul 19 08:41:34.780953 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:34.780914 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9nlzs/perf-node-gather-daemonset-p72vg" Jul 19 08:41:34.970834 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:34.970549 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9nlzs/perf-node-gather-daemonset-p72vg"] Jul 19 08:41:35.206505 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:35.206462 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9nlzs/perf-node-gather-daemonset-p72vg" event={"ID":"9ef09a6d-8f13-4f49-801e-6d29053f187e","Type":"ContainerStarted","Data":"6b1f1dc433c5ae27fd2816fcd2e812f9c5049d500df55df23df800de9febd370"} Jul 19 08:41:35.206742 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:35.206556 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9nlzs/perf-node-gather-daemonset-p72vg" event={"ID":"9ef09a6d-8f13-4f49-801e-6d29053f187e","Type":"ContainerStarted","Data":"5d96a9fc93b4539148a0e55ac57dd7de47ced3719f59d1568f650654e3f7e2a5"} Jul 19 08:41:35.206742 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:35.206587 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-9nlzs/perf-node-gather-daemonset-p72vg" Jul 19 08:41:35.222297 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:35.222230 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-9nlzs/perf-node-gather-daemonset-p72vg" podStartSLOduration=1.222204939 podStartE2EDuration="1.222204939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-19 08:41:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:41:35.220287097 +0000 UTC m=+3561.574596699" watchObservedRunningTime="2026-07-19 08:41:35.222204939 +0000 UTC m=+3561.576514541" Jul 19 08:41:35.242830 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:35.242800 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wmhfd_dd68d31d-f763-433a-867b-65b12eabac76/console-operator/1.log" Jul 19 08:41:35.248508 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:35.248484 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wmhfd_dd68d31d-f763-433a-867b-65b12eabac76/console-operator/2.log" Jul 19 08:41:36.285714 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:36.285683 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78cf4c6f6-rtvvb_d562d83b-2730-4701-b48b-43b8dc8651cc/volume-data-source-validator/0.log" Jul 19 08:41:37.298772 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:37.298742 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-l7s85_4821058e-2da0-4086-8911-6b394fb44223/dns/0.log" Jul 19 08:41:37.316929 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:37.316877 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-l7s85_4821058e-2da0-4086-8911-6b394fb44223/kube-rbac-proxy/0.log" Jul 19 08:41:37.373416 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:37.373374 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:41:37.377452 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:37.377418 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5dh7h_cecbc684-8b2f-4c00-a7ac-2966f0d95873/dns-node-resolver/0.log" Jul 19 08:41:37.477912 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:37.477873 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:41:37.967218 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:37.967184 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ctt95_67825d1c-90c9-4e7f-87c8-575b38afa13a/node-ca/0.log" Jul 19 08:41:39.040160 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:39.040129 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-56f59f876d-qqxn9_1c5887e0-25ef-48da-a06c-a1874a64e2c7/kube-auth-proxy/0.log" Jul 19 08:41:39.185697 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:39.185661 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5669bd84dd-j7hs7_f56affaf-b38d-4ea4-ac8c-309edaa8852e/router/0.log" Jul 19 08:41:39.869576 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:39.869538 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-m78pm_33646489-0313-4d23-96c3-4b32f50233af/serve-healthcheck-canary/0.log" Jul 19 08:41:40.364172 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:40.364143 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-555d9d7885-xr7x7_0adaffaf-8ab7-4966-abd1-5bbd8a0ba882/insights-operator/0.log" Jul 19 08:41:40.365717 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:40.365690 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-555d9d7885-xr7x7_0adaffaf-8ab7-4966-abd1-5bbd8a0ba882/insights-operator/1.log" Jul 19 08:41:40.459358 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:40.459329 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-m7frd_83634718-f9cd-428d-a341-f61661f5bec3/kube-rbac-proxy/0.log" Jul 19 08:41:40.477540 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:40.477498 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-m7frd_83634718-f9cd-428d-a341-f61661f5bec3/exporter/0.log" Jul 19 08:41:40.495728 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:40.495703 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-m7frd_83634718-f9cd-428d-a341-f61661f5bec3/extractor/0.log" Jul 19 08:41:41.227633 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:41.227596 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-9nlzs/perf-node-gather-daemonset-p72vg" Jul 19 08:41:42.898192 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:42.898144 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-gdc7l_33b278bc-869f-4d53-9c01-78e3252c1fc8/manager/0.log" Jul 19 08:41:43.101888 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:43.101847 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-lvdn2_a402f5d0-4bcc-45be-9d79-13c7e577d3c3/server/0.log" Jul 19 08:41:43.182320 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:43.182188 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-85fwf_d389a635-9a6d-42d5-b70c-b78a0898b929/manager/0.log" Jul 19 08:41:43.328967 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:43.328930 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-789467ff8c-j7mlr_6e217fee-5171-40e1-82fb-155e4a0431d1/manager/0.log" Jul 19 08:41:43.344561 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:43.344508 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-789467ff8c-j7mlr_6e217fee-5171-40e1-82fb-155e4a0431d1/copy-manifests/0.log" Jul 19 08:41:44.747956 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:44.747871 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bnnwq_8259d1a2-204c-4d43-a9ce-939a685dc44f/konnectivity-agent/0.log" limit=1024 Jul 19 08:41:45.539097 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:45.539066 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-56f59f876d-qqxn9_1c5887e0-25ef-48da-a06c-a1874a64e2c7/kube-auth-proxy/0.log" Jul 19 08:41:45.655550 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:45.655487 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5669bd84dd-j7hs7_f56affaf-b38d-4ea4-ac8c-309edaa8852e/router/0.log" Jul 19 08:41:46.979775 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:46.979744 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-56f59f876d-qqxn9_1c5887e0-25ef-48da-a06c-a1874a64e2c7/kube-auth-proxy/0.log" Jul 19 08:41:47.097689 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:47.097646 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5669bd84dd-j7hs7_f56affaf-b38d-4ea4-ac8c-309edaa8852e/router/0.log" Jul 19 08:41:49.692235 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:49.692206 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-lxg2w_d4372f5b-2bef-4b86-a076-8b89ef6445c9/migrator/0.log" Jul 19 08:41:49.710292 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:49.710252 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-lxg2w_d4372f5b-2bef-4b86-a076-8b89ef6445c9/graceful-termination/0.log" Jul 19 08:41:50.041984 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:50.041950 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7658d9b749-2hx2z_fc2e627b-03e6-48c5-a476-00eaa505368a/kube-storage-version-migrator-operator/1.log" Jul 19 08:41:50.042857 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:50.042830 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7658d9b749-2hx2z_fc2e627b-03e6-48c5-a476-00eaa505368a/kube-storage-version-migrator-operator/0.log" Jul 19 08:41:51.214713 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:51.214683 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lpbgg_6d024bd7-3f48-4487-b740-2bb4a4e197fc/kube-multus-additional-cni-plugins/0.log" Jul 19 08:41:51.234799 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:51.234771 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lpbgg_6d024bd7-3f48-4487-b740-2bb4a4e197fc/egress-router-binary-copy/0.log" Jul 19 08:41:51.253644 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:51.253613 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lpbgg_6d024bd7-3f48-4487-b740-2bb4a4e197fc/cni-plugins/0.log" Jul 19 08:41:51.271502 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:51.271475 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lpbgg_6d024bd7-3f48-4487-b740-2bb4a4e197fc/bond-cni-plugin/0.log" Jul 19 08:41:51.292829 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:51.292803 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lpbgg_6d024bd7-3f48-4487-b740-2bb4a4e197fc/routeoverride-cni/0.log" Jul 19 08:41:51.310923 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:51.310898 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lpbgg_6d024bd7-3f48-4487-b740-2bb4a4e197fc/whereabouts-cni-bincopy/0.log" Jul 19 08:41:51.329029 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:51.329001 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lpbgg_6d024bd7-3f48-4487-b740-2bb4a4e197fc/whereabouts-cni/0.log" Jul 19 08:41:51.508741 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:51.508661 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-k76h2_dba80f1e-6d8a-47c2-a0dc-034f93da275b/kube-multus/0.log" Jul 19 08:41:51.530558 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:51.530508 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-4r4zh_8279f51a-80ab-4d6f-98b0-32805b4d0355/network-metrics-daemon/0.log" Jul 19 08:41:51.546330 ip-10-0-136-70 kubenswrapper[2574]: I0719 08:41:51.546301 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-4r4zh_8279f51a-80ab-4d6f-98b0-32805b4d0355/kube-rbac-proxy/0.log"